2 Commits

Author	SHA1	Message	Date
Dawid Jurkiewicz	0bba61bbcd	Fix checking if response is a binary string. ... Modyfiy Makefile - enlarge to 40 parallel crawles. Add 4XX http code to retry list. Remove processed.final.txt Probably fix remove_blacklisted.py	2018-04-13 21:45:20 +02:00
Dawid Jurkiewicz	21ba56a8fa	Add domain-blacklist.txt, domain filter, modify crawler. ... Add binary or not checker.	2018-04-09 23:53:36 +02:00