tzietkiewicz/aitech-ium
4
1
Fork 1
Code Issues Pull Requests Releases Wiki Activity

Merge pull request 'Updated DVC remote config' (#33) from tzietkiewicz/aitech-ium:master into master

Browse Source 
Reviewed-on: AITech/aitech-ium#33
This commit is contained in:
Tomasz Zietkiewicz 2022-05-29 10:23:08 +02:00
commit 9be417dd6a
1 changed files with 142 additions and 82 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

224
IUM_10.DVC.ipynb
View File

@ -519,7 +519,7 @@
},
{
"cell_type": "code",
"execution_count": 12,
"execution_count": 1,
"id": "aae59ec2",
"metadata": {
"slideshow": {
@ -533,7 +533,7 @@
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": 1,
"id": "1e522a93",
"metadata": {
"slideshow": {
@ -545,7 +545,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"/home/tomek/AITech/repo/aitech-ium-private/IUM_10/sample-ml-project\n"
"/home/tomek/repos/aitech-ium/IUM_10/sample-ml-project\n"
]
}
],
@ -568,7 +568,7 @@
},
{
"cell_type": "code",
"execution_count": 17,
"execution_count": 2,
"id": "c13c525b",
"metadata": {
"slideshow": {
@ -580,7 +580,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Initialized empty Git repository in /home/tomek/AITech/repo/aitech-ium-private/IUM_10/sample-ml-project/.git/\r\n"
"Reinitialized existing Git repository in /home/tomek/repos/aitech-ium/IUM_10/sample-ml-project/.git/\r\n"
]
}
],
@ -602,7 +602,7 @@
},
{
"cell_type": "code",
"execution_count": 18,
"execution_count": 75,
"id": "44f28226",
"metadata": {
"slideshow": {
@ -654,7 +654,7 @@
},
{
"cell_type": "code",
"execution_count": 19,
"execution_count": 4,
"id": "d1aefe16",
"metadata": {
"slideshow": {
@ -674,12 +674,6 @@
" (use \"git rm --cached <file>...\" to unstage)\r\n",
"\t\u001b[32mnew file: .dvc/.gitignore\u001b[m\r\n",
"\t\u001b[32mnew file: .dvc/config\u001b[m\r\n",
"\t\u001b[32mnew file: .dvc/plots/confusion.json\u001b[m\r\n",
"\t\u001b[32mnew file: .dvc/plots/confusion_normalized.json\u001b[m\r\n",
"\t\u001b[32mnew file: .dvc/plots/default.json\u001b[m\r\n",
"\t\u001b[32mnew file: .dvc/plots/linear.json\u001b[m\r\n",
"\t\u001b[32mnew file: .dvc/plots/scatter.json\u001b[m\r\n",
"\t\u001b[32mnew file: .dvc/plots/smooth.json\u001b[m\r\n",
"\t\u001b[32mnew file: .dvcignore\u001b[m\r\n",
"\r\n"
]
@ -703,9 +697,10 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 6,
"id": "59780e99",
"metadata": {
"scrolled": true,
"slideshow": {
"slide_type": "slide"
}
@ -715,8 +710,11 @@
"name": "stdout",
"output_type": "stream",
"text": [
"On branch master\r\n",
"nothing to commit, working tree clean\r\n"
"[master (root-commit) d00d0ac] Initial commit\r\n",
" 3 files changed, 6 insertions(+)\r\n",
" create mode 100644 .dvc/.gitignore\r\n",
" create mode 100644 .dvc/config\r\n",
" create mode 100644 .dvcignore\r\n"
]
}
],
@ -726,7 +724,7 @@
},
{
"cell_type": "markdown",
"id": "dd8e529b",
"id": "3d4ce1cb",
"metadata": {
"slideshow": {
"slide_type": "slide"
@ -738,7 +736,18 @@
" - wydajnością\n",
" - przestrzenią w repozytorium\n",
" - ograniczenia ze strony serwisu (np. [limit 100 MB na plik w Github](https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-large-files-on-github))\n",
" - Git posiada rozszerzenie [lfs(Large File Storage)](https://git-lfs.github.com/), które stanowi pewne rozwiązanie tego problemu. Same pliki przechowywane są na specjalnym zdalnym serwerze, w repozytorium przechowywane są jedynie odnośniki do tych plików i pewne metadane\n",
" - Git posiada rozszerzenie [lfs(Large File Storage)](https://git-lfs.github.com/), które stanowi pewne rozwiązanie tego problemu. Same pliki przechowywane są na specjalnym zdalnym serwerze, w repozytorium przechowywane są jedynie odnośniki do tych plików i pewne metadane"
]
},
{
"cell_type": "markdown",
"id": "dd8e529b",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
" - DVC proponuje podobne podejście, ale:\n",
" - pliki mogą być przechowywane na niemal dowolnym serwerze, również lokalnie\n",
" - brak limitu wielkości plików (w Git-LFS na Github [limit 2GB](https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-git-large-file-storage))\n",
@ -760,7 +769,7 @@
},
{
"cell_type": "code",
"execution_count": 19,
"execution_count": 9,
"id": "f05ece1b",
"metadata": {
"slideshow": {
@ -772,9 +781,10 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Downloading iris.zip to /home/tomek/AITech/repo/aitech-ium-private/IUM_10/sample-ml-project\n",
"Warning: Your Kaggle API key is readable by other users on this system! To fix this, you can run 'chmod 600 /home/tomek/.kaggle/kaggle.json'\n",
"Downloading iris.zip to /home/tomek/repos/aitech-ium/IUM_10/sample-ml-project\n",
" 0%| | 0.00/3.60k [00:00<?, ?B/s]\n",
"100%|██████████████████████████████████████| 3.60k/3.60k [00:00<00:00, 2.63MB/s]\n",
"100%|██████████████████████████████████████| 3.60k/3.60k [00:00<00:00, 1.66MB/s]\n",
"Archive: iris.zip\n",
" inflating: Iris.csv \n",
" inflating: database.sqlite \n"
@ -803,7 +813,7 @@
},
{
"cell_type": "code",
"execution_count": 14,
"execution_count": 10,
"id": "74d182c7",
"metadata": {
"slideshow": {
@ -815,14 +825,30 @@
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[2K\u001b[32m⠧\u001b[0m Checking graph \u001b[32m⠋\u001b[0m Checking graph\n",
"Adding... \n",
"!\u001b[A\n",
" 0%| |.E8dZEGBYoRayYsJLdesNS4.tmp 0.00/5.11k [00:00<?, ?it/s]\u001b[A\n",
"100% Add|██████████████████████████████████████████████|1/1 [00:04, 4.71s/file]\u001b[A\n",
" 0% Checking cache in '/home/tomek/repos/aitech-ium/IUM_10/sample-ml-project/.d\u001b[A\n",
" \u001b[A\n",
"!\u001b[A\n",
" 0%| |Transferring 0/1 [00:00<?, ?file/s]\u001b[A\n",
" \u001b[A\n",
"!\u001b[A\n",
" 0%| |.oAL9GSGErYepJSZTnvkTL8.tmp 0.00/? [00:00<?, ?B/s]\u001b[A\n",
" 0%| |.oAL9GSGErYepJSZTnvkTL8.tmp 0.00/4.00 [00:00<?, ?B/s]\u001b[A\n",
" \u001b[A\n",
"!\u001b[A\n",
" 0%| |7820ef0af287ff346c5cabfb4c612c 0.00/? [00:00<?, ?B/s]\u001b[A\n",
" 0%| |7820ef0af287ff346c5cabfb4c612c 0.00/4.99k [00:00<?, ?B/s]\u001b[A\n",
"100% Adding...|████████████████████████████████████████|1/1 [00:00, 2.59file/s]\u001b[A\n",
"\n",
"To track the changes with git, run:\n",
"\n",
"\tgit add data/Iris.csv.dvc data/.gitignore\n",
" git add data/.gitignore data/Iris.csv.dvc\n",
"\n",
"To enable auto staging, run:\n",
"\n",
"\tdvc config core.autostage true\n",
"\u001b[0m"
]
}
@ -846,7 +872,7 @@
},
{
"cell_type": "code",
"execution_count": 16,
"execution_count": 11,
"id": "74d54652",
"metadata": {
"slideshow": {
@ -863,7 +889,6 @@
" (use \"git add <file>...\" to include in what will be committed)\r\n",
"\t\u001b[31mdata/.gitignore\u001b[m\r\n",
"\t\u001b[31mdata/Iris.csv.dvc\u001b[m\r\n",
"\t\u001b[31miris.zip\u001b[m\r\n",
"\r\n",
"nothing added to commit but untracked files present (use \"git add\" to track)\r\n"
]
@ -887,7 +912,7 @@
},
{
"cell_type": "code",
"execution_count": 21,
"execution_count": 12,
"id": "460c4a17",
"metadata": {
"slideshow": {
@ -901,7 +926,7 @@
},
{
"cell_type": "code",
"execution_count": 22,
"execution_count": 13,
"id": "80644077",
"metadata": {
"slideshow": {
@ -913,10 +938,10 @@
"name": "stdout",
"output_type": "stream",
"text": [
"[master cc0821a] Dodano dane IRIS (DVC)\r\n",
" 2 files changed, 5 insertions(+)\r\n",
" create mode 100644 data/.gitignore\r\n",
" create mode 100644 data/Iris.csv.dvc\r\n"
"[master 67214ea] Dodano dane IRIS (DVC)\n",
" 2 files changed, 5 insertions(+)\n",
" create mode 100644 data/.gitignore\n",
" create mode 100644 data/Iris.csv.dvc\n"
]
}
],
@ -968,7 +993,7 @@
},
{
"cell_type": "code",
"execution_count": 27,
"execution_count": 16,
"id": "1d471f3a",
"metadata": {
"slideshow": {
@ -981,7 +1006,7 @@
"output_type": "stream",
"text": [
"total 8\r\n",
"-r--r--r-- 1 tomek tomek 5107 wrz 19 2019 7820ef0af287ff346c5cabfb4c612c\r\n"
"-r--r--r-- 1 tomek tomek 5107 Sep 19 2019 7820ef0af287ff346c5cabfb4c612c\r\n"
]
}
],
@ -991,8 +1016,8 @@
},
{
"cell_type": "code",
"execution_count": 1,
"id": "9666e0ae",
"execution_count": 17,
"id": "f86a5b55",
"metadata": {
"slideshow": {
"slide_type": "slide"
@ -1003,7 +1028,9 @@
"name": "stdout",
"output_type": "stream",
"text": [
"ls: cannot access './data': No such file or directory\r\n"
"total 8\r\n",
"-rw-r--r-- 1 tomek tomek 5107 May 29 09:19 Iris.csv\r\n",
"-rw-r--r-- 1 tomek tomek 76 May 29 09:19 Iris.csv.dvc\r\n"
]
}
],
@ -1029,8 +1056,12 @@
},
{
"cell_type": "markdown",
"id": "5e47dbd3",
"metadata": {},
"id": "53429521",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"Obsługiwane typy zdalnych lokalizacji (remotes): https://dvc.org/doc/command-reference/remote/add#supported-storage-types\n",
" - Amazon S3\n",
@ -1049,8 +1080,8 @@
},
{
"cell_type": "code",
"execution_count": 28,
"id": "731f6ea4",
"execution_count": 71,
"id": "a16f2bfa",
"metadata": {
"slideshow": {
"slide_type": "slide"
@ -1061,7 +1092,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Setting 'my_local_remote' as a default remote.\r\n",
"Setting 'my_local_remote' as a default remote.\n",
"\u001b[0m"
]
}
@ -1072,7 +1103,7 @@
},
{
"cell_type": "code",
"execution_count": 39,
"execution_count": 25,
"id": "9c3deeaf",
"metadata": {
"slideshow": {
@ -1085,12 +1116,7 @@
"output_type": "stream",
"text": [
"On branch master\r\n",
"Changes not staged for commit:\r\n",
" (use \"git add <file>...\" to update what will be committed)\r\n",
" (use \"git restore <file>...\" to discard changes in working directory)\r\n",
"\t\u001b[31mmodified: .dvc/config\u001b[m\r\n",
"\r\n",
"no changes added to commit (use \"git add\" and/or \"git commit -a\")\r\n"
"nothing to commit, working tree clean\r\n"
]
}
],
@ -1100,7 +1126,7 @@
},
{
"cell_type": "code",
"execution_count": 41,
"execution_count": 26,
"id": "899eac7d",
"metadata": {
"slideshow": {
@ -1112,8 +1138,8 @@
"name": "stdout",
"output_type": "stream",
"text": [
"[master 3ff62b6] Added DVC remote\r\n",
" 1 file changed, 4 insertions(+)\r\n"
"On branch master\r\n",
"nothing to commit, working tree clean\r\n"
]
}
],
@ -1137,21 +1163,34 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 28,
"id": "c7f24f75",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" 0% Transferring| |0/1 [00:00<?, ?file/s]\n",
"!\u001b[A\n",
" 0%| |7820ef0af287ff346c5cabfb4c612c 0.00/? [00:00<?, ?B/s]\u001b[A\n",
" 0%| |7820ef0af287ff346c5cabfb4c612c 0.00/4.99k [00:00<?, ?B/s]\u001b[A\n",
"1 file pushed \u001b[A\n",
"\u001b[0m"
]
}
],
"source": [
"!dvc push"
]
},
{
"cell_type": "code",
"execution_count": 33,
"execution_count": 31,
"id": "8a355575",
"metadata": {
"slideshow": {
@ -1163,8 +1202,8 @@
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[34;42m/dvcstore\u001b[00m\r\n",
"└── \u001b[01;34m71\u001b[00m\r\n",
"\u001b[34;42m/dvcstore\u001b[0m\r\n",
"└── \u001b[01;34m71\u001b[0m\r\n",
" └── 7820ef0af287ff346c5cabfb4c612c\r\n",
"\r\n",
"1 directory, 1 file\r\n"
@ -1204,7 +1243,7 @@
},
{
"cell_type": "code",
"execution_count": 37,
"execution_count": 32,
"id": "dde39796",
"metadata": {
"slideshow": {
@ -1218,7 +1257,7 @@
},
{
"cell_type": "code",
"execution_count": 42,
"execution_count": 33,
"id": "7f14ec60",
"metadata": {
"slideshow": {
@ -1241,7 +1280,7 @@
},
{
"cell_type": "code",
"execution_count": 43,
"execution_count": 34,
"id": "8a841039",
"metadata": {
"slideshow": {
@ -1266,7 +1305,7 @@
},
{
"cell_type": "code",
"execution_count": 44,
"execution_count": 35,
"id": "bf6c1067",
"metadata": {
"slideshow": {
@ -1278,14 +1317,30 @@
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[2K\u001b[32m⠹\u001b[0m Checking graph \u001b[32m⠋\u001b[0m Checking graph\n",
"Adding... \n",
"!\u001b[A\n",
" 0%| |.TatTHknArFHCT9iDCtxHzh.tmp 0.00/5.07k [00:00<?, ?it/s]\u001b[A\n",
"100% Add|██████████████████████████████████████████████|1/1 [00:00, 2.68file/s]\u001b[A\n",
" 0% Checking cache in '/home/tomek/repos/aitech-ium/IUM_10/sample-ml-project/.d\u001b[A\n",
" \u001b[A\n",
"!\u001b[A\n",
" 0%| |Transferring 0/1 [00:00<?, ?file/s]\u001b[A\n",
" \u001b[A\n",
"!\u001b[A\n",
" 0%| |.GbNyfXVqWGYkQKjqaSP8tL.tmp 0.00/? [00:00<?, ?B/s]\u001b[A\n",
" 0%| |.GbNyfXVqWGYkQKjqaSP8tL.tmp 0.00/4.00 [00:00<?, ?B/s]\u001b[A\n",
" \u001b[A\n",
"!\u001b[A\n",
" 0%| |cff2e578d76852294184c1dce9fdbf 0.00/? [00:00<?, ?B/s]\u001b[A\n",
" 0%| |cff2e578d76852294184c1dce9fdbf 0.00/4.95k [00:00<?, ?B/s]\u001b[A\n",
"100% Adding...|████████████████████████████████████████|1/1 [00:00, 11.00file/s]\u001b[A\n",
"\n",
"To track the changes with git, run:\n",
"\n",
"\tgit add data/Iris.csv.dvc\n",
" git add data/Iris.csv.dvc\n",
"\n",
"To enable auto staging, run:\n",
"\n",
"\tdvc config core.autostage true\n",
"\u001b[0m"
]
}
@ -1296,7 +1351,7 @@
},
{
"cell_type": "code",
"execution_count": 46,
"execution_count": 36,
"id": "4a4865c9",
"metadata": {
"slideshow": {
@ -1308,7 +1363,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"[master e38c244] Removed last line from Iris dataset\r\n",
"[master d6ff265] Removed last line from Iris dataset\r\n",
" 1 file changed, 2 insertions(+), 2 deletions(-)\r\n"
]
}
@ -1642,7 +1697,7 @@
},
"source": [
"### Lokalnie\n",
"Będziemy potrzebować zależności :\n",
"Będziemy potrzebować zależności ([szczegóły](https://dvc.org/doc/command-reference/remote/add))\n",
" \n",
" `conda install dvc-ssh` \n",
"\n",
@ -1733,7 +1788,7 @@
},
{
"cell_type": "code",
"execution_count": 27,
"execution_count": 65,
"id": "e9a04876",
"metadata": {
"slideshow": {
@ -1751,12 +1806,12 @@
}
],
"source": [
"!dvc remote add -f -d ium_ssh_remote ssh://ium-sftp@tzietkiewicz.vm.wmi.amu.edu.pl/ium-sftp"
"!dvc remote add -f -d ium_ssh_remote ssh://ium-sftp@tzietkiewicz.vm.wmi.amu.edu.pl"
]
},
{
"cell_type": "code",
"execution_count": 28,
"execution_count": 66,
"id": "e3f27bbb",
"metadata": {
"slideshow": {
@ -1769,7 +1824,7 @@
"output_type": "stream",
"text": [
"my_local_remote\t/dvcstore\n",
"ium_ssh_remote\tssh://ium-sftp@tzietkiewicz.vm.wmi.amu.edu.pl/ium-sftp\n",
"ium_ssh_remote\tssh://ium-sftp@tzietkiewicz.vm.wmi.amu.edu.pl\n",
"\u001b[0m"
]
}
@ -1778,9 +1833,17 @@
"!dvc remote list"
]
},
{
"cell_type": "markdown",
"id": "c92edd7b",
"metadata": {},
"source": [
"Zapisujemy hasło:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": 67,
"id": "5b2fa175",
"metadata": {
"slideshow": {
@ -1792,17 +1855,17 @@
"name": "stdout",
"output_type": "stream",
"text": [
"/bin/bash: dvc: command not found\r\n"
"\u001b[0m"
]
}
],
"source": [
"!dvc remote modify --local ium_ssh_remote password #[hasło takie jak do serwera MLflow ]"
"!dvc remote modify --local ium_ssh_remote password IUM@2021"
]
},
{
"cell_type": "code",
"execution_count": 30,
"execution_count": 68,
"id": "ea6e16fa",
"metadata": {
"slideshow": {
@ -1814,10 +1877,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
" 0% Uploading| |0/1 [00:00<?, ?file/s]\n",
"!\u001b[A\n",
" 0%| |data/Iris.csv 0.00/4.95k [00:00<?, ?B/s]\u001b[A\n",
"1 file pushed \u001b[A\n",
"Everything is up to date. \n",
"\u001b[0m"
]
}
@ -1877,7 +1937,7 @@
"celltoolbar": "Slideshow",
"email": "tomasz.zietkiewicz@amu.edu.pl",
"kernelspec": {
"display_name": "Python 3",
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
@ -1892,7 +1952,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.9"
"version": "3.10.4"
},
"slideshow": {
"slide_type": "slide"