mass-scraper/parishwebsites/generate_spider_commands.sh
Dawid Jurkiewicz 8b72d0b351 Prototype rule based masses extractor.
Added spider.
Started working on testsets.
2018-03-01 14:40:13 +01:00

6 lines
203 B
Bash
Executable File

#!/usr/bin/env bash
while IFS='$\n' read -r url; do
echo "scrapy crawl parishes -a url=\"$url\" -t jsonlines -o data/`echo "$url" | sed -Ee 's@/|:@@g' | sed 's/^http//g' | sed 's/^www\.//g'`"
done