s396341/mass-scraper
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
30 Commits 1 Branch 0 Tags 32 MiB
Python 74.1%
HTML 17.8%
Makefile 3.5%
Shell 3.5%
Org 1.1%
0bba61bbcd
Go to file
Dawid Jurkiewicz 0bba61bbcd Fix checking if response is a binary string. 
Modyfiy Makefile - enlarge to 40 parallel crawles.
Add 4XX http code to retry list.
Remove processed.final.txt
Probably fix remove_blacklisted.py
2018-04-13 21:45:20 +02:00
extractor Prototype rule based masses extractor. 2018-03-01 14:40:13 +01:00
parishwebsites Fix checking if response is a binary string. 2018-04-13 21:45:20 +02:00
scraper Modifiy error logging in get_parishes_url. Enhance crawl_deon.py 2018-04-06 23:33:18 +02:00
.gitignore Initial commit 2017-03-10 16:05:59 +01:00
environment.yml Add domain-blacklist.txt, domain filter, modify crawler. 2018-04-09 23:53:36 +02:00
LICENSE Initial commit 2017-03-10 16:05:59 +01:00
Makefile Fix checking if response is a binary string. 2018-04-13 21:45:20 +02:00
plan.org Add prototype basic crawl 2017-11-21 22:51:09 +01:00
prepare-environment.sh Switch to pure html download. Enhanced urls filtering. 2018-03-11 18:02:31 +01:00
README.md Update README.md 2018-04-06 23:43:14 +02:00
temat.md Update temat.md 2017-03-14 17:11:33 +01:00

README.md

mass-scraper

Polish masses project. beeminder update