码迷,mamicode.com
首页 > 数据库 > 详细

scrapy中把数据写入mongodb

时间:2019-09-18 09:13:00      阅读:127      评论:0      收藏:0      [点我收藏+]

标签:self   port   setting   imp   rom   span   get   %s   color   

1.setting.py中打开管道

ITEM_PIPELINES = {
   # ‘tianmao.pipelines.TianmaoPipeline‘: 300,
}

2.setting.py中写入mongodb配置

# mongodb
HOST = "127.0.0.1"  # 服务器地址
PORT = 27017  # mongo默认端口号
USER = "用户名"
PWD = "密码"
DB = "数据库名"
TABLE = "表名"

3.pipeline.py文件中倒入pymongo,数据写入数据库

from pymongo import MongoClient
class TianmaoPipeline(object):
    def __init__(self, host, port, user, pwd, db, table):
        self.host = host
        self.port = port
        self.user = user
        self.pwd = pwd
        self.db = db
        self.table = table

    @classmethod
    def from_crawler(cls, crawler):
        HOST = crawler.settings.get(HOST)
        PORT = crawler.settings.get(PORT)
        USER = crawler.settings.get(USER)
        PWD = crawler.settings.get(PWD)
        DB = crawler.settings.get(DB)
        TABLE = crawler.settings.get(TABLE)
        return cls(HOST, PORT, USER, PWD, DB, TABLE)

    def open_spider(self, spider):
        self.client = MongoClient(mongodb://%s:%s@%s:%s %(self.user,self.pwd,self.host,self.port))

    def close_spider(self, spider):
        self.client.close()

    def process_item(self, item, spider):
        self.client[self.db][self.table].save(dict(item))

 

scrapy中把数据写入mongodb

标签:self   port   setting   imp   rom   span   get   %s   color   

原文地址:https://www.cnblogs.com/ptwg/p/11538302.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!