Modyfiy Makefile - enlarge to 40 parallel crawles. Add 4XX http code to retry list. Remove processed.final.txt Probably fix remove_blacklisted.py
Add binary or not checker.