Scrapy Redis: Scaling Your Scrapy Spiders Using Distributed Workers
In this video, we look at how you can scale out and speed up your web scraping using multiple workers(Spiders) using scrapy-redis.
The article that goes along with this video:
https://scrapeops.io/python-scrapy-pl...
Scrapy project used in this tutorial:
https://github.com/python-scrapy-play...
Where you can sign up for a free cloud Redis instance:
https://redis.com/try-free/
Where you can download RedisInsite the Redis desktop manager:
https://redis.com/redis-enterprise/re...
00:00 - Intro
00:37 - Things to think about when scaling
03:01 - Setting up Redis
05:31 - Adding list of urls to be scraped to Redis queue
08:21 - Scraping with one worker (spider)
12:49 - Scraping with two workers (spiders)
16:59 - Outro
Watch video Scrapy Redis: Scaling Your Scrapy Spiders Using Distributed Workers online without registration, duration hours minute second in high quality. This video was added by user ScrapeOps 15 December 2022, don't forget to share it with your friends and acquaintances, it has been viewed on our site 3,560 once and liked it 74 people.