mass-scraper/environment.yml

name: polish-masses
channels:
  - defaults
dependencies:
  - python
  - scrapy
  - gunicorn
  - flask
  - redis-py
  - lxml
  - requests
  - beautifulsoup4
  - colorama
  - pip:
    - tldextract
    - python-google-places
    - jsonlines
    - ipdb
    - html2text
    - binaryornot
Prototype rule based masses extractor. Added spider. Started working on testsets. 2018-01-20 21:55:26 +01:00			`name: polish-masses`
			`channels:`
Add domain-blacklist.txt, domain filter, modify crawler. Add binary or not checker. 2018-04-09 23:52:11 +02:00			`- defaults`
Prototype rule based masses extractor. Added spider. Started working on testsets. 2018-01-20 21:55:26 +01:00			`dependencies:`
			`- python`
			`- scrapy`
Add basic wsgi app. Rename extractors, change directories. Add gunicorn and flask to environment.yml Update .gitignore 2018-04-27 22:44:15 +02:00			`- gunicorn`
			`- flask`
Add test.py for data gathering (data for annotation) Small changes to annotator.py (to be deleted in near future) Add utils/iterator Add redis to enviroment.yml Rename, adapt and move rule based extractor. Adapt find_hours. Yapify webapp app (probalby nothing more) Rename buttons in index.html 2018-05-11 23:12:21 +02:00			`- redis-py`
			`- lxml`
			`- requests`
			`- beautifulsoup4`
			`- colorama`
Prototype rule based masses extractor. Added spider. Started working on testsets. 2018-01-20 21:55:26 +01:00			`- pip:`
			`- tldextract`
			`- python-google-places`
			`- jsonlines`
			`- ipdb`
Switch to pure html download. Enhanced urls filtering. Update Makefile. 2018-03-11 18:02:31 +01:00			`- html2text`
Add domain-blacklist.txt, domain filter, modify crawler. Add binary or not checker. 2018-04-09 23:52:11 +02:00			`- binaryornot`