Scrapy-Redis
https://github.com/rolando/scrapy-redis
pip install scrapy_redis
And just replace the basic settings.py
--You can use redis for scrapy scheduler, start_urls queue, pipeline (separate extended settings) --Easy for external collaboration and multiplexing --Since the scheduler queue is persistent, it can be restarted even if the crawler is stopped in the middle. --You can start multiple same Spiders in multiple processes or multiple servers and crawl in parallel at the same time. --By setting the output to redis in pipeline, the subsequent processing can be processed by another worker process. --Bulk write to DB --Talknize --Stream in machine learning --Spider's start_urls can also be redis, so you can push it from an external service to the start URL queue.
Depending on the tuning, Scrapy can go about 1000 pages / minute with 1 core (100% CPU can be eaten). By using Scrapy-Redis, you can crawl the number of cores * 1000 pages / minute.
Recommended Posts