tzietkiewicz/aitech-ium

Fork 1

Tomasz f7b4bce1d0 Zaktualizowano zajęcia wprowadzające i organizacyjne

2023-03-08 21:46:47 +01:00

95 KiB

Raw Blame History

Logo 1

Inżynieria uczenia maszynowego

10. DVC [laboratoria]

Tomasz Ziętkiewicz (2022)

Logo 2

DVC - Data Version Control

dvc.org
"Version Control System for Machine Learning Projects" (System kontroli wersji dla projektów uczenia maszynowego)
Open Source
Umożliwia:
- wersjonowanie danych i modeli. "Git dla danych i modeli"
- budowanie potoków ("pipeline") definiujących jak budować/trenować/ewaluować modele. "Makefile dla uczenia maszynowego"
- śledzenie, porównywanie metryk i parametrów
ściśle zintegowany z gitem
działa niezależnie od używanego języka/bibliotek i systemu operacyjnego
5-minutowe wprowadzenie: https://www.youtube.com/watch?v=UbL7VUpv1Bs&t=197s

Instalacja i inicjalizacja

https://dvc.org/doc/install
pip(x) install dvc albo:
conda install dvc

!pip3 install dvc

Collecting dvc
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/96/9d/9cca62742cb99da8002c9d6ac6c7463344deb60a219ca4f3a9778b02c67b/dvc-2.8.1-py3-none-any.whl (386kB)
[K    100% |████████████████████████████████| 389kB 1.4MB/s ta 0:00:01
[?25hCollecting setuptools>=34.0.0 (from dvc)
  Using cached https://files.pythonhosted.org/packages/b0/3a/88b210db68e56854d0bcf4b38e165e03be377e13907746f825790f3df5bf/setuptools-59.6.0-py3-none-any.whl
Collecting colorama>=0.3.9 (from dvc)
  Using cached https://files.pythonhosted.org/packages/44/98/5b86278fbbf250d239ae0ecb724f8572af1c91f4a11edf4d36a206189440/colorama-0.4.4-py2.py3-none-any.whl
Collecting dictdiffer>=0.8.1 (from dvc)
  Downloading https://files.pythonhosted.org/packages/47/ef/4cb333825d10317a36a1154341ba37e6e9c087bac99c1990ef07ffdb376f/dictdiffer-0.9.0-py2.py3-none-any.whl
Collecting ply>=3.9 (from dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/a3/58/35da89ee790598a0700ea49b2a66594140f44dec458c07e8e3d4979137fc/ply-3.11-py2.py3-none-any.whl (49kB)
[K    100% |████████████████████████████████| 51kB 1.8MB/s ta 0:00:01
[?25hCollecting nanotime>=0.5.2 (from dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/d5/54/6d5924f59cf671326e7809f4b3f70fa8df535d67e952ad0b6fea02f52faf/nanotime-0.5.2.tar.gz
Collecting fsspec[http]>=2021.10.0 (from dvc)
  Downloading https://files.pythonhosted.org/packages/f6/90/32e53b96067954c2f916667f5a11634aeabef8ed70e83133ed8037b8111b/fsspec-2022.1.0-py3-none-any.whl (133kB)
[K    100% |████████████████████████████████| 143kB 1.3MB/s ta 0:00:01
[?25hCollecting networkx>=2.5 (from dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/f3/b7/c7f488101c0bb5e4178f3cde416004280fd40262433496830de8a8c21613/networkx-2.5.1-py3-none-any.whl (1.6MB)
[K    100% |████████████████████████████████| 1.6MB 609kB/s ta 0:00:01
[?25hCollecting flufl.lock<4,>=3.2 (from dvc)
  Downloading https://files.pythonhosted.org/packages/1e/68/393c148df629f90a919de653ebb967a8bd8c83d07d2bc3150ca0faff3940/flufl.lock-3.2.tar.gz
Collecting shtab<2,>=1.3.4 (from dvc)
  Downloading https://files.pythonhosted.org/packages/37/c0/7d887589bb87db6b87b55315ac41acbbe7355074006a0968343d52ded5cd/shtab-1.5.4-py2.py3-none-any.whl
Collecting shortuuid>=0.5.0 (from dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/01/14/df1aa61e1bb75a6fff19b25d7d175b7f188dc15a70dfc80e7badd5bda1de/shortuuid-1.0.9-py3-none-any.whl
Collecting gitpython>3 (from dvc)
  Using cached https://files.pythonhosted.org/packages/55/60/f884f01eef2a7255875862ec1b12d57d74113ec6e8d9e16c4d254cd6aa3c/GitPython-3.1.20-py3-none-any.whl
Collecting tqdm<5,>=4.45.0 (from dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/8a/c4/d15f1e627fff25443ded77ea70a7b5532d6371498f9285d44d62587e209c/tqdm-4.64.0-py2.py3-none-any.whl (78kB)
[K    100% |████████████████████████████████| 81kB 2.0MB/s ta 0:00:01
[?25hCollecting zc.lockfile>=1.2.1 (from dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/6c/2a/268389776288f0f26c7272c70c36c96dcc0bdb88ab6216ea18e19df1fadd/zc.lockfile-2.0-py2.py3-none-any.whl
Collecting rich>=10.9.0 (from dvc)
  Downloading https://files.pythonhosted.org/packages/05/c0/844c879d659e15c4b1fbe8be89b25de7cf9b0d5e8485aadfe83ae6e7286a/rich-12.4.1-py3-none-any.whl (231kB)
[K    100% |████████████████████████████████| 235kB 1.5MB/s ta 0:00:01
[?25hCollecting aiohttp-retry>=2.4.5 (from dvc)
  Downloading https://files.pythonhosted.org/packages/0f/fe/58130a432d4397e174b82e57d8a11ccf5066631d03949dada4aac8b88041/aiohttp_retry-2.4.6-py3-none-any.whl
Collecting voluptuous>=0.11.7 (from dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/a7/68/927add5dfd55a0d666ffc8939ff4390b76ca3ffbc36c12369f9a034393cb/voluptuous-0.13.1-py3-none-any.whl
Collecting psutil>=5.8.0 (from dvc)
  Downloading https://files.pythonhosted.org/packages/d6/de/0999ea2562b96d7165812606b18f7169307b60cd378bc29cf3673322c7e9/psutil-5.9.1.tar.gz (479kB)
[K    100% |████████████████████████████████| 481kB 1.2MB/s ta 0:00:011
[?25hCollecting flatten-dict<1,>=0.4.1 (from dvc)
  Downloading https://files.pythonhosted.org/packages/43/f5/ee39c6e92acc742c052f137b47c210cd0a1b72dcd3f98495528bb4d27761/flatten_dict-0.4.2-py2.py3-none-any.whl
Collecting configobj>=5.0.6 (from dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/64/61/079eb60459c44929e684fa7d9e2fdca403f67d64dd9dbac27296be2e0fab/configobj-5.0.6.tar.gz
Collecting pygtrie>=2.3.2 (from dvc)
  Downloading https://files.pythonhosted.org/packages/a5/8b/90d0f21a27a354e808a73eb0ffb94db990ab11ad1d8b3db3e5196c882cad/pygtrie-2.4.2.tar.gz
Collecting tabulate>=0.8.7 (from dvc)
  Downloading https://files.pythonhosted.org/packages/ca/80/7c0cad11bd99985cfe7c09427ee0b4f9bd6b048bd13d4ffb32c6db237dfb/tabulate-0.8.9-py3-none-any.whl
Collecting dulwich>=0.20.23 (from dvc)
  Downloading https://files.pythonhosted.org/packages/50/88/d5f6b478086a78995a71151a956ac97d6635f0f6d80f625c96ba13cc612d/dulwich-0.20.40.tar.gz (423kB)
[K    100% |████████████████████████████████| 430kB 1.1MB/s ta 0:00:01
[?25hCollecting appdirs>=1.4.3 (from dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/3b/00/2344469e2084fb287c2e0b57b72910309874c3245463acd6cf5e3db69324/appdirs-1.4.4-py2.py3-none-any.whl
Collecting ruamel.yaml>=0.17.11 (from dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/9e/cb/938214ac358fbef7058343b3765c79a1b7ed0c366f7f992ce7ff38335652/ruamel.yaml-0.17.21-py3-none-any.whl (109kB)
[K    100% |████████████████████████████████| 112kB 2.1MB/s ta 0:00:01
[?25hCollecting python-benedict>=0.24.2 (from dvc)
  Downloading https://files.pythonhosted.org/packages/77/84/6268f9c31140f23ef290c89cb91197f5ea6cbbf102a15db13c1eea845a22/python_benedict-0.25.1-py3-none-any.whl (41kB)
[K    100% |████████████████████████████████| 51kB 2.2MB/s ta 0:00:01
[?25hCollecting grandalf==0.6 (from dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/54/f4/a0b6a4c6d616d0a838b2dd0bc7bf74d73e8e8cdc880bab7fdb5fdc3d0e06/grandalf-0.6-py3-none-any.whl
Collecting pyparsing==2.4.7 (from dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl (67kB)
[K    100% |████████████████████████████████| 71kB 2.1MB/s ta 0:00:01
[?25hCollecting dpath<3,>=2.0.2 (from dvc)
  Downloading https://files.pythonhosted.org/packages/9d/68/19b2a579121446ba80aa3574f04de6931ad549a22cd5588e43c8031de7f8/dpath-2.0.6-py3-none-any.whl
Collecting pygit2<1.7,>=1.5.0; python_version < "3.7" (from dvc)
  Downloading https://files.pythonhosted.org/packages/6b/23/a8c5b726a58282fe2cadcc63faaddd4be147c3c8e0bd38b233114adf98fd/pygit2-1.6.1.tar.gz (258kB)
[K    100% |████████████████████████████████| 266kB 3.5MB/s eta 0:00:01
[?25hCollecting dataclasses>=0.7; python_version < "3.7" (from dvc)
  Using cached https://files.pythonhosted.org/packages/fe/ca/75fac5856ab5cfa51bbbcefa250182e50441074fdc3f803f6e76451fab43/dataclasses-0.8-py3-none-any.whl
Collecting pyasn1>=0.4.1 (from dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/62/1e/a94a8d635fa3ce4cfc7f506003548d0a2447ae76fd5ca53932970fe3053f/pyasn1-0.4.8-py2.py3-none-any.whl (77kB)
[K    100% |████████████████████████████████| 81kB 2.2MB/s ta 0:00:01
[?25hCollecting pathspec<0.9.0,>=0.6.0 (from dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/29/29/a465741a3d97ea3c17d21eaad4c64205428bde56742360876c4391f930d4/pathspec-0.8.1-py2.py3-none-any.whl
Collecting contextvars>=2.1; python_version < "3.7" (from dvc)
  Downloading https://files.pythonhosted.org/packages/83/96/55b82d9f13763be9d672622e1b8106c85acb83edd7cc2fa5bc67cd9877e9/contextvars-2.4.tar.gz
Collecting distro>=1.3.0 (from dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/e1/54/d08d1ad53788515392bec14d2d6e8c410bffdc127780a9a4aa8e6854d502/distro-1.7.0-py3-none-any.whl
Collecting packaging>=19.0 (from dvc)
  Using cached https://files.pythonhosted.org/packages/05/8e/8de486cbd03baba4deef4142bd643a3e7bbe954a784dc1bb17142572d127/packaging-21.3-py3-none-any.whl
Collecting typing-extensions>=3.7.4 (from dvc)
  Using cached https://files.pythonhosted.org/packages/45/6b/44f7f8f1e110027cf88956b59f2fad776cca7e1704396d043f89effd3a0e/typing_extensions-4.1.1-py3-none-any.whl
Collecting importlib-metadata>=1.4; python_version < "3.8" (from dvc)
  Using cached https://files.pythonhosted.org/packages/a0/a1/b153a0a4caf7a7e3f15c2cd56c7702e2cf3d89b1b359d1f1c5e59d68f4ce/importlib_metadata-4.8.3-py3-none-any.whl
Collecting diskcache>=5.2.1 (from dvc)
  Downloading https://files.pythonhosted.org/packages/a1/c4/80d38cf6852ba87f8b506a91f18b3a485c668f452700689add960d7e2ecc/diskcache-5.4.0-py3-none-any.whl (44kB)
[K    100% |████████████████████████████████| 51kB 2.1MB/s ta 0:00:01
[?25hCollecting pydot>=1.2.4 (from dvc)
  Downloading https://files.pythonhosted.org/packages/ea/76/75b1bb82e9bad3e3d656556eaa353d8cd17c4254393b08ec9786ac8ed273/pydot-1.4.2-py2.py3-none-any.whl
Collecting funcy>=1.14 (from dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/cd/1c/895001e29f870c4625a90af00895ef9c9f4f37b0a9b967d2ed810b7be0fc/funcy-1.17-py2.py3-none-any.whl
Collecting toml>=0.10.1 (from dvc)
  Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl
Collecting requests>=2.22.0 (from dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/2d/61/08076519c80041bc0ffa1a8af0cbd3bf3e2b62af10435d269a9d0f40564d/requests-2.27.1-py2.py3-none-any.whl (63kB)
[K    100% |████████████████████████████████| 71kB 2.2MB/s ta 0:00:01
[?25hCollecting aiohttp; extra == "http" (from fsspec[http]>=2021.10.0->dvc)
  Downloading https://files.pythonhosted.org/packages/f6/3b/2e3b8a5b19cdceb532c61d83077a09afe1f120cb876fb771b0ce577cc0ea/aiohttp-3.8.1-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1MB)
[K    100% |████████████████████████████████| 1.1MB 729kB/s ta 0:00:01
[?25hCollecting decorator<5,>=4.3 (from networkx>=2.5->dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/ed/1b/72a1821152d07cf1d8b6fce298aeb06a7eb90f4d6d41acec9861e7cc6df0/decorator-4.4.2-py2.py3-none-any.whl
Collecting atpublic (from flufl.lock<4,>=3.2->dvc)
  Downloading https://files.pythonhosted.org/packages/ab/3d/3df1468805427fedcf880da42fa26353feea3a31b5a0cc71008adcfdb816/atpublic-2.3.tar.gz
Collecting gitdb<5,>=4.0.1 (from gitpython>3->dvc)
  Using cached https://files.pythonhosted.org/packages/a3/7c/5d747655049bfbf75b5fcec57c8115896cb78d6fafa84f6d3ef4c0f13a98/gitdb-4.0.9-py3-none-any.whl
Collecting importlib-resources; python_version < "3.7" (from tqdm<5,>=4.45.0->dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/24/1b/33e489669a94da3ef4562938cd306e8fa915e13939d7b8277cb5569cb405/importlib_resources-5.4.0-py3-none-any.whl
Collecting commonmark<0.10.0,>=0.9.0 (from rich>=10.9.0->dvc)
  Downloading https://files.pythonhosted.org/packages/b1/92/dfd892312d822f36c55366118b95d914e5f16de11044a27cf10a7d71bbbf/commonmark-0.9.1-py2.py3-none-any.whl (51kB)
[K    100% |████████████████████████████████| 51kB 1.6MB/s ta 0:00:01
[?25hCollecting pygments<3.0.0,>=2.6.0 (from rich>=10.9.0->dvc)
  Using cached https://files.pythonhosted.org/packages/5c/8e/1d9017950034297fffa336c72e693a5b51bbf85141b24a763882cf1977b5/Pygments-2.12.0-py3-none-any.whl
Collecting six<2.0,>=1.12 (from flatten-dict<1,>=0.4.1->dvc)
  Using cached https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl
Collecting urllib3>=1.24.1 (from dulwich>=0.20.23->dvc)
  Downloading https://files.pythonhosted.org/packages/ec/03/062e6444ce4baf1eac17a6a0ebfe36bb1ad05e1df0e20b110de59c278498/urllib3-1.26.9-py2.py3-none-any.whl (138kB)
[K    100% |████████████████████████████████| 143kB 1.7MB/s ta 0:00:01
[?25hCollecting certifi (from dulwich>=0.20.23->dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/11/dd/e015f3780f42dd9af62cf0107b44ea1298926627ecd70c17b0e484e95bcd/certifi-2022.5.18.1-py3-none-any.whl (155kB)
[K    100% |████████████████████████████████| 163kB 1.8MB/s ta 0:00:01
[?25hCollecting ruamel.yaml.clib>=0.2.6; platform_python_implementation == "CPython" and python_version < "3.11" (from ruamel.yaml>=0.17.11->dvc)
  Downloading https://files.pythonhosted.org/packages/2a/25/5b1dfc832ef3b83576c546d1fb3e27f136022cdd1008aab290a1e28ef220/ruamel.yaml.clib-0.2.6-cp36-cp36m-manylinux1_x86_64.whl (552kB)
[K    100% |████████████████████████████████| 552kB 1.2MB/s ta 0:00:01
[?25hCollecting python-slugify<7.0.0,>=6.0.1 (from python-benedict>=0.24.2->dvc)
  Downloading https://files.pythonhosted.org/packages/c1/35/74ab800f1108b95ff9b8e7672a01dbf1f357159e6d06c1f16e983674ff0c/python_slugify-6.1.2-py2.py3-none-any.whl
Collecting python-fsutil<1.0.0,>=0.6.0 (from python-benedict>=0.24.2->dvc)
  Downloading https://files.pythonhosted.org/packages/d3/67/3bceac53a29c2cf6a27ca9a940c7a059af13b5b4283d09368e9c0d7f91b4/python_fsutil-0.6.1-py3-none-any.whl
Collecting mailchecker<5.0.0,>=4.1.0 (from python-benedict>=0.24.2->dvc)
  Downloading https://files.pythonhosted.org/packages/1a/11/49e85a526dd3f2cb6bdf1dc519d64395bd56a88d9b2a5f6f3acd1f8c3f51/mailchecker-4.1.17.tar.gz (232kB)
[K    100% |████████████████████████████████| 235kB 2.7MB/s ta 0:00:011
[?25hCollecting pyyaml<7.0,>=6.0 (from python-benedict>=0.24.2->dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/b3/85/79b9e5b4e8d3c0ac657f4e8617713cca8408f6cdc65d2ee6554217cedff1/PyYAML-6.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (603kB)
[K    100% |████████████████████████████████| 604kB 1.1MB/s ta 0:00:01
[?25hCollecting xmltodict<1.0.0,>=0.12.0 (from python-benedict>=0.24.2->dvc)
  Downloading https://files.pythonhosted.org/packages/94/db/fd0326e331726f07ff7f40675cd86aa804bfd2e5016c727fa761c934990e/xmltodict-0.13.0-py2.py3-none-any.whl
Collecting phonenumbers<9.0.0,>=8.12.0 (from python-benedict>=0.24.2->dvc)
  Downloading https://files.pythonhosted.org/packages/a1/c8/23c7185f589da0edf4f9831de1e718989073aa094780f8ee34e6c29b7a2e/phonenumbers-8.12.48-py2.py3-none-any.whl (2.6MB)
[K    100% |████████████████████████████████| 2.6MB 434kB/s ta 0:00:011
[?25hCollecting ftfy<7.0.0,>=6.0.0 (from python-benedict>=0.24.2->dvc)
  Downloading https://files.pythonhosted.org/packages/af/da/d215a091986e5f01b80f5145cff6f22e2dc57c6b048aab2e882a07018473/ftfy-6.0.3.tar.gz (64kB)
[K    100% |████████████████████████████████| 71kB 2.0MB/s ta 0:00:01
[?25hCollecting python-dateutil<3.0.0,>=2.8.0 (from python-benedict>=0.24.2->dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
  Using cached https://files.pythonhosted.org/packages/36/7a/87837f39d0296e723bb9b62bbb257d0355c7f6128853c78955f57342a56d/python_dateutil-2.8.2-py2.py3-none-any.whl
Collecting future (from grandalf==0.6->dvc)
[33m  Cache entry deserialization failed, entry ignored[0m
[33m  Cache entry deserialization failed, entry ignored[0m
  Downloading https://files.pythonhosted.org/packages/45/0b/38b06fd9b92dc2b68d58b75f900e97884c45bedd2ff83203d933cf5851c9/future-0.18.2.tar.gz (829kB)
[K    100% |████████████████████████████████| 829kB 1.5MB/s eta 0:00:01
[?25hCollecting cffi>=1.4.0 (from pygit2<1.7,>=1.5.0; python_version < "3.7"->dvc)
  Using cached https://files.pythonhosted.org/packages/49/7b/449daf9cacfd7355cea1b4106d2be614315c29ac16567e01756167f6daab/cffi-1.15.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Collecting cached-property (from pygit2<1.7,>=1.5.0; python_version < "3.7"->dvc)
  Downloading https://files.pythonhosted.org/packages/48/19/f2090f7dad41e225c7f2326e4cfe6fff49e57dedb5b53636c9551f86b069/cached_property-1.5.2-py2.py3-none-any.whl
Collecting immutables>=0.9 (from contextvars>=2.1; python_version < "3.7"->dvc)
  Downloading https://files.pythonhosted.org/packages/ff/88/9c71337193c3d24c2cf3c14d5ed05eeb502f9f21fa6117edfa9b3b43bff1/immutables-0.18-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (115kB)
[K    100% |████████████████████████████████| 122kB 2.3MB/s ta 0:00:01
[?25hCollecting zipp>=0.5 (from importlib-metadata>=1.4; python_version < "3.8"->dvc)
  Using cached https://files.pythonhosted.org/packages/bd/df/d4a4974a3e3957fd1c1fa3082366d7fff6e428ddb55f074bf64876f8e8ad/zipp-3.6.0-py3-none-any.whl
Collecting idna<4,>=2.5; python_version >= "3" (from requests>=2.22.0->dvc)
  Downloading https://files.pythonhosted.org/packages/04/a2/d918dcd22354d8958fe113e1a3630137e0fc8b44859ade3063982eacd2a4/idna-3.3-py3-none-any.whl (61kB)
[K    100% |████████████████████████████████| 61kB 1.7MB/s ta 0:00:01
[?25hCollecting charset-normalizer~=2.0.0; python_version >= "3" (from requests>=2.22.0->dvc)
  Downloading https://files.pythonhosted.org/packages/06/b3/24afc8868eba069a7f03650ac750a778862dc34941a4bebeb58706715726/charset_normalizer-2.0.12-py3-none-any.whl
Collecting asynctest==0.13.0; python_version < "3.8" (from aiohttp; extra == "http"->fsspec[http]>=2021.10.0->dvc)
  Downloading https://files.pythonhosted.org/packages/e8/b6/8d17e169d577ca7678b11cd0d3ceebb0a6089a7f4a2de4b945fe4b1c86db/asynctest-0.13.0-py3-none-any.whl
Collecting yarl<2.0,>=1.0 (from aiohttp; extra == "http"->fsspec[http]>=2021.10.0->dvc)
  Downloading https://files.pythonhosted.org/packages/fa/cb/8791922f5ec97b9ebec516d062c0e113da963568ffe2c7c04d8187ab7cc3/yarl-1.7.2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (270kB)
[K    100% |████████████████████████████████| 276kB 1.9MB/s ta 0:00:01
[?25hCollecting idna-ssl>=1.0; python_version < "3.7" (from aiohttp; extra == "http"->fsspec[http]>=2021.10.0->dvc)
  Downloading https://files.pythonhosted.org/packages/46/03/07c4894aae38b0de52b52586b24bf189bb83e4ddabfe2e2c8f2419eec6f4/idna-ssl-1.1.0.tar.gz
Collecting frozenlist>=1.1.1 (from aiohttp; extra == "http"->fsspec[http]>=2021.10.0->dvc)
  Downloading https://files.pythonhosted.org/packages/51/3f/f67395ff0090b9f2835838a1f61c3e840baac70fd65bae762095dead48b2/frozenlist-1.2.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (191kB)
[K    100% |████████████████████████████████| 194kB 1.2MB/s ta 0:00:01
[?25hCollecting async-timeout<5.0,>=4.0.0a3 (from aiohttp; extra == "http"->fsspec[http]>=2021.10.0->dvc)
  Downloading https://files.pythonhosted.org/packages/d6/c1/8991e7c5385b897b8c020cdaad718c5b087a6626d1d11a23e1ea87e325a7/async_timeout-4.0.2-py3-none-any.whl
Collecting aiosignal>=1.1.2 (from aiohttp; extra == "http"->fsspec[http]>=2021.10.0->dvc)
  Downloading https://files.pythonhosted.org/packages/3b/87/fe94898f2d44a93a35d5aa74671ed28094d80753a1113d68b799fab6dc22/aiosignal-1.2.0-py3-none-any.whl
Collecting attrs>=17.3.0 (from aiohttp; extra == "http"->fsspec[http]>=2021.10.0->dvc)
  Using cached https://files.pythonhosted.org/packages/be/be/7abce643bfdf8ca01c48afa2ddf8308c2308b0c3b239a44e57d020afa0ef/attrs-21.4.0-py2.py3-none-any.whl
Collecting multidict<7.0,>=4.5 (from aiohttp; extra == "http"->fsspec[http]>=2021.10.0->dvc)
  Downloading https://files.pythonhosted.org/packages/82/43/81ddfbcfbdfaeaa0624f36dcb715dc8135562377b3292e93b0315a861e92/multidict-5.2.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (159kB)
[K    100% |████████████████████████████████| 163kB 2.0MB/s ta 0:00:01
[?25hCollecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->gitpython>3->dvc)
  Using cached https://files.pythonhosted.org/packages/6d/01/7caa71608bc29952ae09b0be63a539e50d2484bc37747797a66a60679856/smmap-5.0.0-py3-none-any.whl
Collecting text-unidecode>=1.3 (from python-slugify<7.0.0,>=6.0.1->python-benedict>=0.24.2->dvc)
  Downloading https://files.pythonhosted.org/packages/a6/a5/c0b6468d3824fe3fde30dbb5e1f687b291608f9473681bbf7dabbf5a87d7/text_unidecode-1.3-py2.py3-none-any.whl (78kB)
[K    100% |████████████████████████████████| 81kB 2.2MB/s ta 0:00:01
[?25hCollecting wcwidth (from ftfy<7.0.0,>=6.0.0->python-benedict>=0.24.2->dvc)
  Using cached https://files.pythonhosted.org/packages/59/7c/e39aca596badaf1b78e8f547c807b04dae603a433d3e7a7e04d67f2ef3e5/wcwidth-0.2.5-py2.py3-none-any.whl
Collecting pycparser (from cffi>=1.4.0->pygit2<1.7,>=1.5.0; python_version < "3.7"->dvc)
  Using cached https://files.pythonhosted.org/packages/62/d5/5f610ebe421e85889f2e55e33b7f9a6795bd982198517d912eb1c76e1a53/pycparser-2.21-py2.py3-none-any.whl
Building wheels for collected packages: nanotime, flufl.lock, psutil, configobj, pygtrie, dulwich, pygit2, contextvars, atpublic, mailchecker, ftfy, future, idna-ssl
  Running setup.py bdist_wheel for nanotime ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/41/99/17/7135f635215e1f61e906295afd11f4f791cfe4ab45f3bfdca2
  Running setup.py bdist_wheel for flufl.lock ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/4f/51/d7/f65a7b7f37da7594f7021b122fe677187667ad21f1171d2514
  Running setup.py bdist_wheel for psutil ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/6e/94/8f/ef906811f8dcf6824a9747df0381615be48d723073fb59a317
  Running setup.py bdist_wheel for configobj ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/f1/e4/16/4981ca97c2d65106b49861e0b35e2660695be7219a2d351ee0
  Running setup.py bdist_wheel for pygtrie ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/db/57/91/73782136379fe419036c5ec0e4070d8b3a35f2a36bd6a94ed8
  Running setup.py bdist_wheel for dulwich ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/63/28/8c/0bbff7d6e30f3fc523639b000b33aba9155152e9eb23689ba0
  Running setup.py bdist_wheel for pygit2 ... [?25lerror
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-hr8mtrcf/pygit2/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmp4kyicel3pip-wheel- --python-tag cp36:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.6
  creating build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/__init__.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/_build.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/_run.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/blame.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/callbacks.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/config.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/credentials.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/errors.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/ffi.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/index.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/packbuilder.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/refspec.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/remote.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/repository.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/settings.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/submodule.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/utils.py -> build/lib.linux-x86_64-3.6/pygit2
  creating build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/attr.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/blame.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/buffer.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/callbacks.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/checkout.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/clone.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/common.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/config.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/describe.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/diff.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/errors.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/graph.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/index.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/indexer.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/merge.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/net.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/oid.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/pack.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/proxy.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/refspec.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/remote.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/repository.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/revert.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/stash.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/strarray.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/submodule.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/transport.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/types.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  running build_ext
  generating cffi module 'build/temp.linux-x86_64-3.6/pygit2._libgit2.c'
  creating build/temp.linux-x86_64-3.6
  building 'pygit2._pygit2' extension
  creating build/temp.linux-x86_64-3.6/src
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/include -I/usr/include/python3.6m -c src/blob.c -o build/temp.linux-x86_64-3.6/src/blob.o
  In file included from src/blob.c:30:0:
  src/blob.h:33:10: fatal error: git2.h: No such file or directory
   #include <git2.h>
            ^~~~~~~~
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  
  ----------------------------------------
[31m  Failed building wheel for pygit2[0m
[?25h  Running setup.py clean for pygit2
  Running setup.py bdist_wheel for contextvars ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/a5/7d/68/1ebae2668bda2228686e3c1cf16f2c2384cea6e9334ad5f6de
  Running setup.py bdist_wheel for atpublic ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/33/25/82/57d46b60a048f8e30b31f10497539498a3b826c78e2433c2d4
  Running setup.py bdist_wheel for mailchecker ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/fd/e1/e7/804e77a70eac7103bdba2f4b3e1eba36840b38554a4b8152c8
  Running setup.py bdist_wheel for ftfy ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/99/2c/e6/109c8a28fef7a443f67ba58df21fe1d0067ac3322e75e6b0b7
  Running setup.py bdist_wheel for future ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/8b/99/a0/81daf51dcd359a9377b110a8a886b3895921802d2fc1b2397e
  Running setup.py bdist_wheel for idna-ssl ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/d3/00/b3/32d613e19e08a739751dd6bf998cfed277728f8b2127ad4eb7
Successfully built nanotime flufl.lock psutil configobj pygtrie dulwich contextvars atpublic mailchecker ftfy future idna-ssl
Failed to build pygit2
Installing collected packages: setuptools, colorama, dictdiffer, ply, nanotime, asynctest, idna, multidict, typing-extensions, yarl, idna-ssl, frozenlist, async-timeout, aiosignal, charset-normalizer, attrs, aiohttp, certifi, urllib3, requests, fsspec, decorator, networkx, atpublic, flufl.lock, shtab, shortuuid, smmap, gitdb, gitpython, zipp, importlib-resources, tqdm, zc.lockfile, dataclasses, commonmark, pygments, rich, aiohttp-retry, voluptuous, psutil, six, importlib-metadata, flatten-dict, configobj, pygtrie, tabulate, dulwich, appdirs, ruamel.yaml.clib, ruamel.yaml, text-unidecode, python-slugify, python-fsutil, mailchecker, pyyaml, toml, xmltodict, phonenumbers, wcwidth, ftfy, python-dateutil, python-benedict, future, pyparsing, grandalf, dpath, pycparser, cffi, cached-property, pygit2, pyasn1, pathspec, immutables, contextvars, distro, packaging, diskcache, pydot, funcy, dvc
  Running setup.py install for pygit2 ... [?25lerror
    Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-hr8mtrcf/pygit2/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-6tml5js1-record/install-record.txt --single-version-externally-managed --compile --user --prefix=:
    running install
    /home/tomek/.local/lib/python3.6/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      setuptools.SetuptoolsDeprecationWarning,
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/__init__.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/_build.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/_run.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/blame.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/callbacks.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/config.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/credentials.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/errors.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/ffi.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/index.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/packbuilder.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/refspec.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/remote.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/repository.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/settings.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/submodule.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/utils.py -> build/lib.linux-x86_64-3.6/pygit2
    creating build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/attr.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/blame.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/buffer.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/callbacks.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/checkout.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/clone.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/common.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/config.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/describe.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/diff.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/errors.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/graph.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/index.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/indexer.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/merge.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/net.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/oid.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/pack.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/proxy.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/refspec.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/remote.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/repository.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/revert.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/stash.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/strarray.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/submodule.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/transport.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/types.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    running build_ext
    generating cffi module 'build/temp.linux-x86_64-3.6/pygit2._libgit2.c'
    creating build/temp.linux-x86_64-3.6
    building 'pygit2._pygit2' extension
    creating build/temp.linux-x86_64-3.6/src
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/include -I/usr/include/python3.6m -c src/blob.c -o build/temp.linux-x86_64-3.6/src/blob.o
    In file included from src/blob.c:30:0:
    src/blob.h:33:10: fatal error: git2.h: No such file or directory
     #include <git2.h>
              ^~~~~~~~
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    
    ----------------------------------------
[31mCommand "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-hr8mtrcf/pygit2/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-6tml5js1-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-build-hr8mtrcf/pygit2/[0m
[?25h

Stwórzmy katalog, w którym będziemy przechowywać nasz projekt:

!mkdir -p IUM_10/sample-ml-project

#Jupyter notebook magic https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-cd
%cd "IUM_10/sample-ml-project"

/home/tomek/repos/aitech-ium/IUM_10/sample-ml-project

Inicjalizujemy puste repozytorium Git (możemy też pominąć ten krok i działać w istniejącym już repozytorium)

!git init

Reinitialized existing Git repository in /home/tomek/repos/aitech-ium/IUM_10/sample-ml-project/.git/

Teraz inicjalizujemy repozytorium DVC:

!dvc init

[31mERROR[39m: failed to initiate DVC - '.dvc' exists. Use `-f` to force.
[0m

Zobaczmy jakie pliki dodał (również do repozytorium git) DVC. Ich opis znajdziemy tutaj: https://dvc.org/doc/user-guide/project-structure/internal-files

!git status

On branch master
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
	[32mmodified:   .dvc/config[m

Możemy teraz zacommitować zmiany w git:

!git commit -m "Initial commit"

[master (root-commit) d00d0ac] Initial commit
 3 files changed, 6 insertions(+)
 create mode 100644 .dvc/.gitignore
 create mode 100644 .dvc/config
 create mode 100644 .dvcignore

Śledzenie plików za pomocą DVC

dużymi plikami, takimi jak plikami z danymi wejściowymi czy plikami modeli, trudno zarządza się za pomocą gita, ze względu na problemy z:
- wydajnością
- przestrzenią w repozytorium
- ograniczenia ze strony serwisu (np. limit 100 MB na plik w Github)
Git posiada rozszerzenie lfs(Large File Storage), które stanowi pewne rozwiązanie tego problemu. Same pliki przechowywane są na specjalnym zdalnym serwerze, w repozytorium przechowywane są jedynie odnośniki do tych plików i pewne metadane

DVC proponuje podobne podejście, ale:
- pliki mogą być przechowywane na niemal dowolnym serwerze, również lokalnie
- brak limitu wielkości plików (w Git-LFS na Github limit 2GB)
- DVC zapewnia dodatkowe narzędzie umożliwiające śledzenie plików i ich powiązań z wynikami eksperymentów
- więcej, patrz tutaj

Przygotujmy przykładowe dane, pobierając je z Kaggle:

!kaggle datasets download -d uciml/iris
!unzip -o iris.zip
!rm database.sqlite iris.zip
!mkdir -p data
!mv Iris.csv data/

Downloading iris.zip to /home/tomek/repos/aitech-ium/IUM_10/sample-ml-project
  0%|                                               | 0.00/3.60k [00:00<?, ?B/s]
100%|██████████████████████████████████████| 3.60k/3.60k [00:00<00:00, 2.10MB/s]
Archive:  iris.zip
  inflating: Iris.csv                
  inflating: database.sqlite

Teraz dodamy plik(i) z danymi do DVC:

!dvc add data/Iris.csv

[2K[32m⠋[0m Checking graph                                                   [32m⠋[0m Checking graph
Adding...                                                                       
![A
  0% Checking cache in '/home/tomek/repos/aitech-ium/IUM_10/sample-ml-project/.d[A
                                                                                [A
![A
  0%|          |Transferring                          0/1 [00:00<?,     ?file/s][A
                                                                                [A
![A
  0%|          |.ZfWsiEaSUu83Mc7QqhDoTS.tmp        0.00/? [00:00<?,        ?B/s][A
  0%|          |.ZfWsiEaSUu83Mc7QqhDoTS.tmp     0.00/4.00 [00:00<?,        ?B/s][A
                                                                                [A
![A
  0%|          |7820ef0af287ff346c5cabfb4c612c     0.00/? [00:00<?,        ?B/s][A
  0%|          |7820ef0af287ff346c5cabfb4c612c 0.00/4.99k [00:00<?,        ?B/s][A
100% Adding...|████████████████████████████████████████|1/1 [00:00,  5.92file/s][A

To track the changes with git, run:

    git add data/Iris.csv.dvc

To enable auto staging, run:

	dvc config core.autostage true
[0m

DVC utworzył plik data/Iris.csv.dvc i dadał oryginalny plik do .gitignore
W repozytorium będzie obecny tylko plik *.dvc, zawierający odnośnik do prawdziwego pliku

!git status -u

On branch master
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
	[32mmodified:   .dvc/config[m

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	[31mmodified:   data/Iris.csv.dvc[m

Dodajmy pliki data/Iris.csv.dvc data/.gitignore do repozytorium git, zgodnie z sugestią DVC:

!git add data/Iris.csv.dvc data/.gitignore

!git commit -m "Dodano dane IRIS (DVC)"

[master b37a8d8] Dodano dane IRIS (DVC)
 2 files changed, 2 insertions(+), 6 deletions(-)

Plik *.dvc zawiera m.in. hash pliku. Więcej o plikach *.dvc: link

# %load data/Iris.csv.dvc
outs:
- md5: 717820ef0af287ff346c5cabfb4c612c
  size: 5107
  path: Iris.csv

Oryginalny plik Iris.csv został przeniesiony do katalogu ./dvc/cache/{wartość hash pliku) i podlinkowany z powrotem do oryginalnej lokalizacji. Sposób tworzenia linków może być różny w zależności od systemu plików.

!ls -l .dvc/cache/71

total 8
-r--r--r-- 1 tomek tomek 5107 Sep 19  2019 7820ef0af287ff346c5cabfb4c612c

!ls -l ./data

total 8
-rw-r--r-- 1 tomek tomek 5107 May 29 09:19 Iris.csv
-rw-r--r-- 1 tomek tomek   76 May 29 09:19 Iris.csv.dvc

dvc remote

żeby wysłać właściwe pliki śledzone przez DVC do zdalnej lokalizacji (z której będą mogłby być pobrane np. przez system CI albo innych użytkowników) musimy mieć skonfigurowaną taką lokazliację
służy do tego polecenie dvc remote add
użyjemy lokalnego "remote". Tutaj będzie to po prostu utworzony wcześniej katalog /dvcstore. Taki katalog istnieje też na naszym Jenkinsie, oczywiście należy go podmontować w Dockerze
w realnych zastosowaniach podalibyśmy tutaj ścieżkę do jakiegoś zasobu dostępnego przez inernet jak np. serwer SFTP, ścieżka do AWS S3 itp.

Obsługiwane typy zdalnych lokalizacji (remotes): https://dvc.org/doc/command-reference/remote/add#supported-storage-types

Amazon S3
S3-compatible storage
Microsoft Azure Blob Storage
Google Drive
Google Cloud Storage
Aliyun OSS
SSH
HDFS
WebHDFS
HTTP
WebDAV
local remote

!dvc remote add -d my_local_remote /dvcstore

Setting 'my_local_remote' as a default remote.
[0m

!git status

On branch master
nothing to commit, working tree clean

!git add .dvc/config
!git commit -m "Added DVC remote"

On branch master
nothing to commit, working tree clean

dvc push

Kiedy mamy już skonfigurowany "remote" możemy wypchnąć do niego pliki korzystając z polecenia dvc push:

!dvc push

  0% Transferring|                                   |0/1 [00:00<?,     ?file/s]
![A
  0%|          |7820ef0af287ff346c5cabfb4c612c     0.00/? [00:00<?,        ?B/s][A
  0%|          |7820ef0af287ff346c5cabfb4c612c 0.00/4.99k [00:00<?,        ?B/s][A
1 file pushed                                                                   [A
[0m

!tree /dvcstore

[34;42m/dvcstore[0m
└── [01;34m71[0m
    └── 7820ef0af287ff346c5cabfb4c612c

1 directory, 1 file

dvc pull

Żeby pobrać dane z DVC (np. w innej lokalizacji, przez innego użytkownika), musimy:

sklonować repozytorium git (żeby m.in. pobrać pliki *.dvc
wykonać dvc pull

Dodawanie nowych plików i modyfikacja istniejących wygląda podobnie jak przy zwykłych plikach śledzonych przez git, tylko zamiast git używamy polecenia dvc a dodatkowo pamiętamy o zarządzaniu plikami *.dvc za pomocą gita:

!head -n -1 data/Iris.csv | sponge data/Iris.csv

!git status

On branch master
nothing to commit, working tree clean

!dvc status

data/Iris.csv.dvc:                                                    core[39m>
	changed outs:
		modified:           data/Iris.csv
[0m

!dvc add data/Iris.csv

[2K[32m⠹[0m Checking graph                                                   [32m⠋[0m Checking graph
Adding...                                                                       
![A
  0% Checking cache in '/home/tomek/repos/aitech-ium/IUM_10/sample-ml-project/.d[A
                                                                                [A
![A
  0%|          |Transferring                          0/1 [00:00<?,     ?file/s][A
                                                                                [A
![A
  0%|          |.AquCc93WCb2aAJ98voTeFG.tmp        0.00/? [00:00<?,        ?B/s][A
  0%|          |.AquCc93WCb2aAJ98voTeFG.tmp     0.00/4.00 [00:00<?,        ?B/s][A
                                                                                [A
![A
  0%|          |4705c4d470a4d9dd152808e5e9f56f     0.00/? [00:00<?,        ?B/s][A
  0%|          |4705c4d470a4d9dd152808e5e9f56f 0.00/4.92k [00:00<?,        ?B/s][A
100% Adding...|████████████████████████████████████████|1/1 [00:00, 11.09file/s][A

To track the changes with git, run:

    git add data/Iris.csv.dvc

To enable auto staging, run:

	dvc config core.autostage true
[0m

!git add data/Iris.csv.dvc
!git commit -m "Removed last line from Iris dataset"

[master 5379e3b] Removed last line from Iris dataset
 1 file changed, 2 insertions(+), 2 deletions(-)

dvc checkout

Polecenia dvc checkout używamy razem z git checkout, żeby zmienić branch, na którym pracujemy.
DVC podmieni wersje plików śledzonych przez siebie na pochodzące z innego brancha (o ile pliki te się różnią i różnią się pliki *.dvc w odpowiednich branchach
zmiana brancha przez git powoduje (ewentualną) zmianę plików *.dvc a dvc checkout kopiuje/linkuje pliki z katalogu .dvc/cache o wartościach hash odpowiadających tym z plików *.dvc

Wymiana danych między projektami

za pomocą poleceń dvc import i dvc update możemy dodać i później aktualizować pliki śledzone przez DVC w innym repozytorium

!dvc import https://github.com/iterative/dataset-registry \
             get-started/data.xml -o data/data.xml

Importing 'get-started/data.xml (https://github.com/iterative/dataset-registry)' -> 'data/data.xml'
  0% Downloading|                                    |0/1 [00:00<?,     ?file/s]
![A
  0%|          |get-started/data.xml           0.00/37.9M [00:00<?,       ?it/s][A
  0%|          |get-started/data.xml      64.0k/36.1M [00:00<02:12,     286kB/s][A
  0%|          |get-started/data.xml       128k/36.1M [00:00<01:33,     403kB/s][A
  1%|          |get-started/data.xml       256k/36.1M [00:00<00:57,     658kB/s][A
  1%|          |get-started/data.xml       384k/36.1M [00:00<00:45,     818kB/s][A
  1%|▏         |get-started/data.xml       512k/36.1M [00:00<00:53,     693kB/s][A
  2%|▏         |get-started/data.xml       640k/36.1M [00:01<00:57,     644kB/s][A
  2%|▏         |get-started/data.xml       768k/36.1M [00:01<00:59,     619kB/s][A
  2%|▏         |get-started/data.xml       896k/36.1M [00:01<00:51,     718kB/s][A
  3%|▎         |get-started/data.xml      1.00M/36.1M [00:01<00:55,     666kB/s][A
  3%|▎         |get-started/data.xml      1.12M/36.1M [00:01<00:57,     633kB/s][A
  3%|▎         |get-started/data.xml      1.25M/36.1M [00:02<00:57,     638kB/s][A
  4%|▍         |get-started/data.xml      1.38M/36.1M [00:02<00:52,     698kB/s][A
  4%|▍         |get-started/data.xml      1.50M/36.1M [00:02<00:55,     656kB/s][A
  4%|▍         |get-started/data.xml      1.62M/36.1M [00:02<00:57,     628kB/s][A
  5%|▍         |get-started/data.xml      1.69M/36.1M [00:02<00:58,     618kB/s][A
  5%|▌         |get-started/data.xml      1.81M/36.1M [00:02<00:53,     675kB/s][A
  5%|▌         |get-started/data.xml      1.94M/36.1M [00:03<00:53,     672kB/s][A
  6%|▌         |get-started/data.xml      2.06M/36.1M [00:03<00:55,     642kB/s][A
  6%|▌         |get-started/data.xml      2.12M/36.1M [00:03<00:56,     628kB/s][A
  6%|▌         |get-started/data.xml      2.19M/36.1M [00:03<00:57,     616kB/s][A
  6%|▌         |get-started/data.xml      2.25M/36.1M [00:03<00:58,     606kB/s][A
  7%|▋         |get-started/data.xml      2.38M/36.1M [00:03<00:48,     732kB/s][A
  7%|▋         |get-started/data.xml      2.50M/36.1M [00:04<00:52,     666kB/s][A
  7%|▋         |get-started/data.xml      2.62M/36.1M [00:04<00:55,     636kB/s][A
  8%|▊         |get-started/data.xml      2.75M/36.1M [00:04<00:56,     614kB/s][A
  8%|▊         |get-started/data.xml      2.88M/36.1M [00:04<00:49,     711kB/s][A
  8%|▊         |get-started/data.xml      3.00M/36.1M [00:04<00:52,     663kB/s][A
  9%|▊         |get-started/data.xml      3.12M/36.1M [00:05<00:54,     637kB/s][A
  9%|▉         |get-started/data.xml      3.25M/36.1M [00:05<00:55,     623kB/s][A
  9%|▉         |get-started/data.xml      3.38M/36.1M [00:05<00:48,     710kB/s][A
 10%|▉         |get-started/data.xml      3.50M/36.1M [00:05<00:51,     664kB/s][A
 10%|█         |get-started/data.xml      3.62M/36.1M [00:05<00:45,     751kB/s][A
 10%|█         |get-started/data.xml      3.75M/36.1M [00:05<00:49,     691kB/s][A
 11%|█         |get-started/data.xml      3.88M/36.1M [00:06<00:43,     777kB/s][A
 11%|█         |get-started/data.xml      4.00M/36.1M [00:06<00:47,     705kB/s][A
 11%|█▏        |get-started/data.xml      4.12M/36.1M [00:06<00:42,     790kB/s][A
 12%|█▏        |get-started/data.xml      4.25M/36.1M [00:06<00:46,     716kB/s][A
 12%|█▏        |get-started/data.xml      4.38M/36.1M [00:06<00:44,     749kB/s][A
 12%|█▏        |get-started/data.xml      4.50M/36.1M [00:07<00:45,     734kB/s][A
 13%|█▎        |get-started/data.xml      4.62M/36.1M [00:07<00:40,     810kB/s][A
 13%|█▎        |get-started/data.xml      4.75M/36.1M [00:07<00:42,     773kB/s][A
 13%|█▎        |get-started/data.xml      4.88M/36.1M [00:07<00:41,     795kB/s][A
 14%|█▍        |get-started/data.xml      5.00M/36.1M [00:07<00:37,     870kB/s][A
 14%|█▍        |get-started/data.xml      5.12M/36.1M [00:07<00:34,     932kB/s][A
 15%|█▍        |get-started/data.xml      5.25M/36.1M [00:07<00:35,     916kB/s][A
 15%|█▍        |get-started/data.xml      5.38M/36.1M [00:08<00:35,     898kB/s][A
 15%|█▌        |get-started/data.xml      5.50M/36.1M [00:08<00:33,     962kB/s][A
 16%|█▌        |get-started/data.xml      5.62M/36.1M [00:08<00:33,     949kB/s][A
 16%|█▌        |get-started/data.xml      5.75M/36.1M [00:08<00:31,    1.00MB/s][A
 16%|█▋        |get-started/data.xml      5.88M/36.1M [00:08<00:30,    1.04MB/s][A
 17%|█▋        |get-started/data.xml      6.06M/36.1M [00:08<00:26,    1.19MB/s][A
 17%|█▋        |get-started/data.xml      6.19M/36.1M [00:08<00:26,    1.19MB/s][A
 17%|█▋        |get-started/data.xml      6.31M/36.1M [00:08<00:26,    1.19MB/s][A
 18%|█▊        |get-started/data.xml      6.50M/36.1M [00:08<00:23,    1.31MB/s][A
 18%|█▊        |get-started/data.xml      6.62M/36.1M [00:09<00:23,    1.30MB/s][A
 19%|█▉        |get-started/data.xml      6.81M/36.1M [00:09<00:21,    1.41MB/s][A
 19%|█▉        |get-started/data.xml      7.00M/36.1M [00:09<00:20,    1.48MB/s][A
 20%|█▉        |get-started/data.xml      7.19M/36.1M [00:09<00:19,    1.54MB/s][A
 20%|██        |get-started/data.xml      7.38M/36.1M [00:09<00:18,    1.60MB/s][A
 21%|██        |get-started/data.xml      7.56M/36.1M [00:09<00:18,    1.62MB/s][A
 21%|██▏       |get-started/data.xml      7.75M/36.1M [00:09<00:17,    1.68MB/s][A
 22%|██▏       |get-started/data.xml      7.94M/36.1M [00:09<00:17,    1.70MB/s][A
 22%|██▏       |get-started/data.xml      8.12M/36.1M [00:10<00:17,    1.72MB/s][A
 23%|██▎       |get-started/data.xml      8.38M/36.1M [00:10<00:15,    1.88MB/s][A
 24%|██▎       |get-started/data.xml      8.56M/36.1M [00:10<00:15,    1.84MB/s][A
 24%|██▍       |get-started/data.xml      8.81M/36.1M [00:10<00:14,    1.96MB/s][A
 25%|██▌       |get-started/data.xml      9.06M/36.1M [00:10<00:13,    2.06MB/s][A
 26%|██▌       |get-started/data.xml      9.31M/36.1M [00:10<00:13,    2.14MB/s][A
 27%|██▋       |get-started/data.xml      9.62M/36.1M [00:10<00:11,    2.32MB/s][A
 27%|██▋       |get-started/data.xml      9.88M/36.1M [00:10<00:11,    2.33MB/s][A
 28%|██▊       |get-started/data.xml      10.2M/36.1M [00:10<00:11,    2.46MB/s][A
 29%|██▉       |get-started/data.xml      10.4M/36.1M [00:11<00:10,    2.45MB/s][A
 30%|██▉       |get-started/data.xml      10.8M/36.1M [00:11<00:10,    2.57MB/s][A
 31%|███       |get-started/data.xml      11.1M/36.1M [00:11<00:09,    2.67MB/s][A
 32%|███▏      |get-started/data.xml      11.4M/36.1M [00:11<00:09,    2.84MB/s][A
 33%|███▎      |get-started/data.xml      11.8M/36.1M [00:11<00:08,    2.85MB/s][A
 34%|███▎      |get-started/data.xml      12.1M/36.1M [00:11<00:08,    3.01MB/s][A
 35%|███▍      |get-started/data.xml      12.5M/36.1M [00:11<00:07,    3.12MB/s][A
 36%|███▌      |get-started/data.xml      12.9M/36.1M [00:11<00:07,    3.22MB/s][A
 37%|███▋      |get-started/data.xml      13.2M/36.1M [00:11<00:07,    3.31MB/s][A
 38%|███▊      |get-started/data.xml      13.7M/36.1M [00:12<00:06,    3.49MB/s][A
 39%|███▉      |get-started/data.xml      14.1M/36.1M [00:12<00:06,    3.62MB/s][A
 40%|████      |get-started/data.xml      14.6M/36.1M [00:12<00:06,    3.74MB/s][A
 42%|████▏     |get-started/data.xml      15.0M/36.1M [00:12<00:05,    3.82MB/s][A
 43%|████▎     |get-started/data.xml      15.4M/36.1M [00:12<00:05,    3.97MB/s][A
 44%|████▍     |get-started/data.xml      15.9M/36.1M [00:12<00:05,    4.08MB/s][A
 45%|████▌     |get-started/data.xml      16.4M/36.1M [00:12<00:04,    4.23MB/s][A
 47%|████▋     |get-started/data.xml      17.0M/36.1M [00:12<00:04,    4.44MB/s][A
 48%|████▊     |get-started/data.xml      17.5M/36.1M [00:12<00:04,    4.52MB/s][A
 50%|████▉     |get-started/data.xml      18.1M/36.1M [00:13<00:04,    4.69MB/s][A
 52%|█████▏    |get-started/data.xml      18.6M/36.1M [00:13<00:03,    4.84MB/s][A
 53%|█████▎    |get-started/data.xml      19.2M/36.1M [00:13<00:03,    5.05MB/s][A
 55%|█████▍    |get-started/data.xml      19.8M/36.1M [00:13<00:03,    5.16MB/s][A
 57%|█████▋    |get-started/data.xml      20.4M/36.1M [00:13<00:03,    5.35MB/s][A
 58%|█████▊    |get-started/data.xml      21.1M/36.1M [00:13<00:02,    5.49MB/s][A
 60%|██████    |get-started/data.xml      21.8M/36.1M [00:13<00:02,    5.66MB/s][A
 62%|██████▏   |get-started/data.xml      22.4M/36.1M [00:13<00:02,    5.83MB/s][A
 64%|██████▍   |get-started/data.xml      23.2M/36.1M [00:14<00:02,    6.05MB/s][A
 66%|██████▌   |get-started/data.xml      23.9M/36.1M [00:14<00:02,    6.20MB/s][A
 68%|██████▊   |get-started/data.xml      24.6M/36.1M [00:14<00:01,    6.40MB/s][A
 70%|███████   |get-started/data.xml      25.4M/36.1M [00:14<00:01,    6.51MB/s][A
 72%|███████▏  |get-started/data.xml      26.0M/36.1M [00:14<00:01,    5.75MB/s][A
 74%|███████▎  |get-started/data.xml      26.6M/36.1M [00:14<00:02,    4.26MB/s][A
 75%|███████▍  |get-started/data.xml      27.1M/36.1M [00:14<00:02,    3.53MB/s][A
 76%|███████▌  |get-started/data.xml      27.5M/36.1M [00:15<00:02,    3.26MB/s][A
 77%|███████▋  |get-started/data.xml      27.9M/36.1M [00:15<00:02,    3.00MB/s][A
 78%|███████▊  |get-started/data.xml      28.2M/36.1M [00:15<00:02,    2.95MB/s][A
 79%|███████▉  |get-started/data.xml      28.5M/36.1M [00:15<00:02,    2.91MB/s][A
 80%|███████▉  |get-started/data.xml      28.8M/36.1M [00:15<00:02,    2.88MB/s][A
 81%|████████  |get-started/data.xml      29.1M/36.1M [00:15<00:02,    2.86MB/s][A
 81%|████████▏ |get-started/data.xml      29.4M/36.1M [00:15<00:02,    2.84MB/s][A
 82%|████████▏ |get-started/data.xml      29.8M/36.1M [00:16<00:02,    2.83MB/s][A
 83%|████████▎ |get-started/data.xml      30.1M/36.1M [00:16<00:02,    2.83MB/s][A
 84%|████████▍ |get-started/data.xml      30.4M/36.1M [00:16<00:02,    2.83MB/s][A
 85%|████████▍ |get-started/data.xml      30.7M/36.1M [00:16<00:02,    2.83MB/s][A
 86%|████████▌ |get-started/data.xml      31.0M/36.1M [00:16<00:01,    2.83MB/s][A
 87%|████████▋ |get-started/data.xml      31.3M/36.1M [00:16<00:01,    2.83MB/s][A
 88%|████████▊ |get-started/data.xml      31.6M/36.1M [00:16<00:01,    2.83MB/s][A
 88%|████████▊ |get-started/data.xml      31.9M/36.1M [00:16<00:01,    2.84MB/s][A
 89%|████████▉ |get-started/data.xml      32.2M/36.1M [00:16<00:01,    2.85MB/s][A
 90%|█████████ |get-started/data.xml      32.6M/36.1M [00:17<00:01,    2.85MB/s][A
 91%|█████████ |get-started/data.xml      32.9M/36.1M [00:17<00:01,    2.86MB/s][A
 92%|█████████▏|get-started/data.xml      33.2M/36.1M [00:17<00:01,    2.86MB/s][A
 93%|█████████▎|get-started/data.xml      33.5M/36.1M [00:17<00:00,    2.87MB/s][A
 94%|█████████▎|get-started/data.xml      33.8M/36.1M [00:17<00:00,    2.87MB/s][A
 94%|█████████▍|get-started/data.xml      34.1M/36.1M [00:17<00:00,    2.87MB/s][A
 95%|█████████▌|get-started/data.xml      34.4M/36.1M [00:17<00:00,    2.87MB/s][A
 96%|█████████▌|get-started/data.xml      34.8M/36.1M [00:17<00:00,    2.87MB/s][A
 97%|█████████▋|get-started/data.xml      35.1M/36.1M [00:17<00:00,    2.87MB/s][A
 98%|█████████▊|get-started/data.xml      35.4M/36.1M [00:18<00:00,    2.87MB/s][A
 99%|█████████▉|get-started/data.xml      35.7M/36.1M [00:18<00:00,    2.88MB/s][A
100%|█████████▉|get-started/data.xml      36.0M/36.1M [00:18<00:00,    2.87MB/s][A
                                                                                [A
To track the changes with git, run:

	git add data/.gitignore data/data.xml.dvc
[0m

!dvc status

Data and pipelines are up to date.                                              
[0m

ls -l data

total 37020
-rw-rw-r-- 1 tomek tomek 37891850 maj 31 11:10 data.xml
-rw-rw-r-- 1 tomek tomek      284 maj 31 11:10 data.xml.dvc
-rw-rw-r-- 1 tomek tomek     5072 maj 31 11:01 Iris.csv
-rw-rw-r-- 1 tomek tomek       76 maj 31 11:01 Iris.csv.dvc

# %load data/data.xml.dvc
md5: a7cd139231cc35ed63541ce3829b96db
frozen: true
deps:
- path: get-started/data.xml
  repo:
    url: https://github.com/iterative/dataset-registry
    rev_lock: ba014f40e29670421a67cb1c47543f402348aa13
outs:
- md5: a304afb96060aad90176268345e10355
  size: 37891850
  path: data.xml

DVC pipelines

wprowadzenie: https://youtu.be/71IGzyH95UY
Getting started: https://dvc.org/doc/start/data-pipelines
dvc pipelines pozwala nam zbudować (za pomocą polecenie dvc run) lub zdefiniować (edytując plik dvc.yaml) graf zależności między krokami wykonywanymi w naszym projekcie (takimi jak "przygotowanie danych", "trenowanie", "ewaluacja")
tak zdefiniowany pipeline można potem uruchomić za pomocą polecenia dvc reproduce

Zadania [15pkt]

Zainicjalizuj repozytorium DVC wewnątrz Twojego repozytorium z projektem [1pkt]
Dodaj plik(i) z danymi w Twoim projekcie do DVC [1pkt]
Skonfiguruj remote (dane do konfiguracji podane poniżej) [3pkt]
Stwórz/zdefiniuj i dodaj do repozytorium plik dvc.yaml opisujący kroki wykonywane w Twoim projekcie. Wydziel przynajmniej 2 kroki (np. przygotowanie danych/trenowanie) powiązane ze sobą za pomocą zależności (skorzystaj z materiałów "Getting started", link powyżej) [6pkt]
Stwórz projekt na Jenkinsie (s1233456-dvc), w którym sklonujesz repozytorium, ściągniesz pliki dvc (za pomocą dvc pull) i uruchomisz pipeline (za pomocą dvc reproduce) [4pkt]

SSH remote

Jednym z remote obsługiwanych przez DVC jest SFTP/SSH. W celu jego wykorzystania na serwerze tzietkiewicz.vm.wmi.amu.edu.pl utworzony został użytkownik ium-sftp i skonfigurowany serwer SFTP. Został też dla niego wygenerowany klucz ssh, który został dodany jako "Jenkins credential" (patrz opis konfiguracji na Jenkins poniżej)

Lokalnie

Będziemy potrzebować zależności (szczegóły)

conda install dvc-ssh

albo

pip install dvc[ssh] paramiko

conda install -c conda-forge dvc-ssh

Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/tomek/miniconda3

  added / updated specs:
    - dvc-ssh


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    bcrypt-3.2.0               |   py39h3811e60_1          44 KB  conda-forge
    ca-certificates-2021.5.30  |       ha878542_0         136 KB  conda-forge
    certifi-2021.5.30          |   py39hf3d152e_0         141 KB  conda-forge
    dvc-2.3.0                  |   py39hf3d152e_0         542 KB  conda-forge
    dvc-ssh-2.3.0              |   py39hf3d152e_0           9 KB  conda-forge
    fsspec-2021.5.0            |     pyhd8ed1ab_0          77 KB  conda-forge
    invoke-1.5.0               |     pyhd3deb0d_0         137 KB  conda-forge
    paramiko-2.7.2             |     pyh9f0ad1d_0         135 KB  conda-forge
    pynacl-1.4.0               |   py39h3811e60_2         1.3 MB  conda-forge
    ------------------------------------------------------------
                                           Total:         2.5 MB

The following NEW packages will be INSTALLED:

  bcrypt             conda-forge/linux-64::bcrypt-3.2.0-py39h3811e60_1
  dvc-ssh            conda-forge/linux-64::dvc-ssh-2.3.0-py39hf3d152e_0
  invoke             conda-forge/noarch::invoke-1.5.0-pyhd3deb0d_0
  paramiko           conda-forge/noarch::paramiko-2.7.2-pyh9f0ad1d_0
  pynacl             conda-forge/linux-64::pynacl-1.4.0-py39h3811e60_2

The following packages will be UPDATED:

  ca-certificates                      2020.12.5-ha878542_0 --> 2021.5.30-ha878542_0
  certifi                          2020.12.5-py39hf3d152e_1 --> 2021.5.30-py39hf3d152e_0
  dvc                                  2.1.0-py39hf3d152e_0 --> 2.3.0-py39hf3d152e_0
  fsspec                                 0.9.0-pyhd8ed1ab_2 --> 2021.5.0-pyhd8ed1ab_0



Downloading and Extracting Packages
certifi-2021.5.30    | 141 KB    | ##################################### | 100% 
fsspec-2021.5.0      | 77 KB     | ##################################### | 100% 
dvc-2.3.0            | 542 KB    | ##################################### | 100% 
invoke-1.5.0         | 137 KB    | ##################################### | 100% 
paramiko-2.7.2       | 135 KB    | ##################################### | 100% 
bcrypt-3.2.0         | 44 KB     | ##################################### | 100% 
pynacl-1.4.0         | 1.3 MB    | ##################################### | 100% 
dvc-ssh-2.3.0        | 9 KB      | ##################################### | 100% 
ca-certificates-2021 | 136 KB    | ##################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

Note: you may need to restart the kernel to use updated packages.

!dvc remote add -f -d ium_ssh_remote ssh://ium-sftp@tzietkiewicz.vm.wmi.amu.edu.pl

Setting 'ium_ssh_remote' as a default remote.
[0m

!dvc remote list

ium_ssh_remote	ssh://ium-sftp@tzietkiewicz.vm.wmi.amu.edu.pl
[0m

Zapisujemy hasło:

!dvc remote modify --local ium_ssh_remote password IUM@2021

[0m

!dvc push

  0% Transferring|                                   |0/1 [00:00<?,     ?file/s]
![A
  0%|          |4705c4d470a4d9dd152808e5e9f56f     0.00/? [00:00<?,        ?B/s][A
  0%|          |4705c4d470a4d9dd152808e5e9f56f 0.00/4.92k [00:00<?,        ?B/s][A
1 file pushed                                                                   [A
[0m

Jenkins

W Jenkins można użyć mechanizmu "Credentials", żeby w bezpieczny sposób przekazać hasło albo klucz prywatny.

Takie dane dla użytkownika ium-sftp zostały stworzone na Jenkinsie:

typu ssh key: https://tzietkiewicz.vm.wmi.amu.edu.pl:8080/credentials/store/system/domain/_/credential/48ac7004-216e-4260-abba-1fe5db753e18/
typu "secret text" - zawierający hasło użytkownika ium-shftp: https://tzietkiewicz.vm.wmi.amu.edu.pl:8080/credentials/store/system/domain/_/credential/ium-sftp-password/

Opis używania "Credentials" w Jenkinsfile: https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#for-other-credential-types

Klucza ssh można użyć tak:

withCredentials(
    [sshUserPrivateKey(credentialsId: '48ac7004-216e-4260-abba-1fe5db753e18', keyFileVariable: 'IUM_SFTP_KEY', passphraseVariable: '', usernameVariable: '')]) {
                sh 'dvc remote add -d ium_ssh_remote ssh://ium-sftp@tzietkiewicz.vm.wmi.amu.edu.pl/ium-sftp'
                sh 'dvc remote modify --local ium_ssh_remote keyfile $IUM_SFTP_KEY'
                sh 'dvc pull'}

Secret text tak:

    withCredentials([string(credentialsId: 'ium-sftp-password', variable: 'IUM_SFTP_PASS')]) {
                sh 'dvc remote add -d ium_ssh_remote ssh://ium-sftp@tzietkiewicz.vm.wmi.amu.edu.pl/ium-sftp'
                sh 'dvc remote modify --local ium_ssh_remote password $IUM_SFTP_KEY'
                sh 'dvc pull'
    }

Przykład kongiguracji:

!dvc pull

Everything is up to date.                                                       
[0m

!git remote add origin git@git.wmi.amu.edu.pl:tzietkiewicz/sample-ml-project.git
!git push --set-upstream origin master

error: remote origin already exists.
Enumerating objects: 24, done.
Counting objects: 100% (24/24), done.
Delta compression using up to 4 threads
Compressing objects: 100% (21/21), done.
Writing objects: 100% (24/24), 2.10 KiB | 715.00 KiB/s, done.
Total 24 (delta 4), reused 0 (delta 0), pack-reused 0
remote: . Processing 1 references[K
remote: Processed 1 references in total[K
To git.wmi.amu.edu.pl:tzietkiewicz/sample-ml-project.git
 * [new branch]      master -> master
Branch 'master' set up to track remote branch 'master' from 'origin'.

95 KiB Raw Blame History Unescape Escape