mass-scraper/parishwebsites/generate_spider_commands.sh
Dawid Jurkiewicz c3b86fe5a9 Prototype rule based masses extractor.
Added spider.
Started working on testsets.
2018-01-20 21:55:26 +01:00

6 lines
203 B
Bash
Executable File

#!/usr/bin/env bash
while IFS='$\n' read -r url; do
echo "scrapy crawl parishes -a url=\"$url\" -t jsonlines -o data/`echo "$url" | sed -Ee 's@/|:@@g' | sed 's/^http//g' | sed 's/^www\.//g'`"
done