Redis与Scrapy

Redis与Scrapy

Redis is an open source, BSD licensed, advanced key-value cache and store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs. ——Redis Home Page

1. 安装scrapy-redis模块

• pip install scrapy-redis
• easy_install scrapy-redis

2. 安装和运行Redis

• http://redis.io/download
• 运行Redis：redis-server redis.conf
• 清空缓存：redis-cli flushdb

3. Scrapy配置Redis

• settings.py配置Redis

SCHEDULER = "scrapy_redis.scheduler.Scheduler"
SCHEDULER_PERSIST =True
SCHEDULER_QUEUE_CLASS = ‘scrapy_redis.queue.SpiderPriorityQueue‘
REDIS_URL = None
REDIS_HOST = ‘127.0.0.1‘ #REDIS服务器的ip
REDIS_PORT = 6379 #REDIS服务器端口

• Spider调用Redis

from scrapy_redis.spiders import RedisSpider
class xxx(RedisSpider):
    ..................
    redis_key = ‘username:password‘
    ..................