siulkilulki
e9c4dcd743
Fix generating spiider commands. Add redirected domain appenid to allowed domains. Configure loggers. Add more meta info to *processed.txt Enhance view raw data python jsnoline viewer
7 lines
327 B
Bash
Executable File
7 lines
327 B
Bash
Executable File
#!/usr/bin/env bash
|
|
|
|
while IFS='$\n' read -r url; do
|
|
filename="`echo "$url" | sed -Ee 's@/|:|\?|\!|\*|\(|\)|=|'"'"'|\+|;|,|\@|#|\[|\]|\$|&@@g' | sed 's/^http//g' | sed 's/^www\.//g'`"
|
|
echo "scrapy crawl parishes -a url=\"$url\" -a filename=\"$filename\" -t jsonlines -o \"data/$filename\" 2> \"logs/$filename\" "
|
|
done
|