标签:completed span code 文件 image load main room pip
功能点:如何爬取图片,并保存到本地
爬取网站:斗鱼主播
完整代码:https://files.cnblogs.com/files/bookwed/Douyu.zip
主要代码:
douyu.py
import scrapy import json from Douyu.items import DouyuItem class DouyuSpider(scrapy.Spider): name = ‘douyu‘ allowed_domains = [‘douyucdn.cn‘] base_url = "http://capi.douyucdn.cn/api/v1/getVerticalRoom?limit=20&offset={}" offset = 0 start_urls = [base_url.format(offset)] # 只能返回Request、BaseItem、dict or None def parse(self, response): data_list = json.loads(response.body)["data"] # 如果没有读到数据,则退出,否则继续请求 if len(data_list) == 0: return for data in data_list: item = DouyuItem() item["nickname"] = data["nickname"] item["image_url"] = data["vertical_src"] yield item self.offset += 20 yield scrapy.Request(self.base_url.format(self.offset), callback=self.parse)
pipelines.py
from scrapy.pipelines.images import ImagesPipeline import scrapy import os from Douyu.settings import IMAGES_STORE as images_store # 读取配置文件的信息 # 继承ImagePipeline,实现对图片的操作,要进行settings的设置:IMAGES_STORE(图片保存位置)、USER_AGENT、ROBOTSTXT_OBEY、ITEM_PIPELINES class DouyuPipeline(ImagesPipeline): # 重写方法 def get_media_requests(self, item, info): image_url = item["image_url"] yield scrapy.Request(image_url) # 保存图片时重命名 def item_completed(self, results, item, info): # print(results) # print("*"* 30) # 列表推导式,获取图片的保存路径 image_url = [x["path"] for ok, x in results if ok] # 重命名,由于都是jpg文件,所以直接拼上了 os.rename(images_store+image_url[0], images_store + item["nickname"] + ".jpg")
scrapy爬虫系列之三--爬取图片保存到本地及日志的基本用法
标签:completed span code 文件 image load main room pip
原文地址:https://www.cnblogs.com/bookwed/p/10617860.html