s444415/wikisource-crawler
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

yellow data, crawler fixes, .gitignore added out files

Browse Source
This commit is contained in:
s444415 2023-01-04 20:32:26 +00:00
parent f7ff1acec3
commit a16d660cf7
3 changed files with 1253134 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -1 +1,2 @@
env env
*.out

2
crawler.py
View File

@ -41,7 +41,7 @@ def main(args):
if r.status_code != 200: if r.status_code != 200:
print(r.__dict__) print(r.__dict__)
time.sleep(10) time.sleep(30)
r = requests.get(MAIN_URL + next_page) r = requests.get(MAIN_URL + next_page)
if r.status_code != 200: if r.status_code != 200:
break break

1253132
yellow.tsv Normal file
View File

File diff suppressed because one or more lines are too long