标签：爬虫 python beautifulsoup xpath scrapy
scrapy有一个简单的入门文档，大家可以参考一下，我感觉官方文档是最靠谱的，也是最真实的。
首先我们先创建一个scrapy的项目
scrapy startproject weather
我采用的是ubuntu12.04的系统，建立项目之后主文件夹就会出现一个weather的文件夹。我们可以通过tree来查看文件夹的结构。可以使用sudoapt-get install tree安装。
tree weather
weather ├── scrapy.cfg ├── wea.json ├── weather │ ├── __init__.py │ ├── __init__.pyc │ ├── items.py │ ├── items.pyc │ ├── pipelines.py │ ├── pipelines.py~ │ ├── pipelines.pyc │ ├── settings.py │ ├── settings.pyc │ └── spiders │ ├── __init__.py │ ├── __init__.pyc │ ├── weather_spider1.py │ ├── weather_spider1.pyc │ ├── weather_spider2.py │ ├── weather_spider2.py~ │ ├── weather_spider2.pyc │ └── weather_spider.pyc ├── weather.json └── wea.txt
上面就是我编写过之后的爬虫文件，现在我们新创建一个weathertest来看一下初始的时候文件是什么样的。
weathertest
├── scrapy.cfg
└── weathertest
├── __init__.py
├── items.py
├── pipelines.py
├── settings.py
└── spiders
└── __init__.py
scrapy.cfg:项目的配置文件 weather/:该项目的python模块。之后您将在此加入代码。 weather/items.py:相当于要提取的元素，相当于一个容器 weather/pipelines.py:存文件时或者发送到其他地方可用其编写 weather/settings.py:项目的设置文件. weather/spiders/:放置spider代码的目录.
Item是保存爬取到的数据的容器；其使用方法和python字典类似，并且提供了额外保护机制来避免拼写错误导致的未定义字段错误。
# -*- coding: utf-8 -*-
# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/items.html
import scrapy
class WeatherItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
city = scrapy.Field()
date = scrapy.Field()
dayDesc = scrapy.Field()
dayTemp = scrapy.Field()
pass
之后我们编写今天的爬虫一号，使用xpath分解html中的标签，为了创建一个Spider，您必须继承scrapy.Spider类， 且定义以下三个属性:
1.name:用于区别Spider。该名字必须是唯一的，您不可以为不同的Spider设定相同的名字。
2.start_urls:包含了Spider在启动时进行爬取的url列表。因此，第一个被获取到的页面将是其中之一。后续的URL则从初始的URL获取到的数据中提取。
3.parse()是spider的一个方法。被调用时，每个初始URL完成下载后生成的Response对象将会作为唯一的参数传递给该函数。该方法负责解析返回的数据(responsedata)，提取数据(生成item)以及生成需要进一步处理的URL的Request对象。
import scrapy
from weather.items import WeatherItem
class WeatherSpider(scrapy.Spider):
name = 'weather_spider1'
allowed_domains = ['sina.com.cn']
start_urls = ['http://weather.sina.com.cn/beijing']
def parse(self,response):
item = WeatherItem()
item['city'] = response.xpath("//*[@id='slider_ct_name']/text()").extract()
tenDay = response.xpath('//*[@id="blk_fc_c0_scroll"]');
item['date'] = tenDay.css('p.wt_fc_c0_i_date::text').extract()
item['dayDesc'] = tenDay.css('img.icons0_wt::attr(title)').extract()
item['dayTemp'] = tenDay.css('p.wt_fc_c0_i_temp::text').extract()
return itemScrapy使用了一种基于XPath和CSS表达式机制:Scrapy Selectors。
这里给出XPath表达式的例子及对应的含义:
/html/head/title:选择HTML文档中<head>标签内的<title>元素
/html/head/title/text():选择上面提到的<title>元素的文字
//td:选择所有的<td>元素
//div[@class="mine"]:选择所有具有class="mine"属性的div元素
上边仅仅是几个简单的XPath例子，XPath实际上要比这远远强大的多。如果您想了解的更多，我们推荐 这篇XPath教程。
为了配合XPath，Scrapy除了提供了Selector之外，还提供了方法来避免每次从response中提取数据时生成selector的麻烦。
Selector有四个基本的方法(点击相应的方法可以看到详细的API文档):
xpath():传入xpath表达式，返回该表达式所对应的所有节点的selectorlist列表 。
css():传入CSS表达式，返回该表达式所对应的所有节点的selectorlist列表.
extract():序列化该节点为unicode字符串并返回list。
re():根据传入的正则表达式对数据进行提取，返回unicode字符串list列表。
然后我们就可以编写pipelines.py文件了，如果你只是想保存文件，也可以不编写这个文件，就保持原样即可，运行爬虫的时候再后面加上 -o weather.json
scrapy crawl weather_spider1 -o weather.json
# -*- coding: utf-8 -*-
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
class WeatherPipeline(object):
def __init__(self):
self.file = open('wea.txt','w+')
def process_item(self, item, spider):
city = item['city'][0].encode('utf-8')
self.file.write('city:'+str(city)+'\n\n')
date = item['date']
desc = item['dayDesc']
dayDesc = desc[1::2]
nightDesc = desc[0::2]
dayTemp = item['dayTemp']
weaitem = zip(date,dayDesc,nightDesc,dayTemp)
for i in range(len(weaitem)):
item = weaitem[i]
d = item[0]
dd = item[1]
nd = item[2]
ta = item[3].split('/')
dt = ta[0]
nt = ta[1]
txt = 'date: {0} \t\t day:{1}({2}) \t\t night:{3}({4}) \n\n'.format(
d,
dd.encode('utf-8'),
dt.encode('utf-8'),
nd.encode('utf-8'),
nt.encode('utf-8')
)
self.file.write(txt)
return item
# -*- coding: utf-8 -*-
# Scrapy settings for weather project
#
# For simplicity, this file contains only the most important settings by
# default. All the other settings are documented here:
#
# http://doc.scrapy.org/en/latest/topics/settings.html
#
BOT_NAME = 'weather'
SPIDER_MODULES = ['weather.spiders']
NEWSPIDER_MODULE = 'weather.spiders'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'weather (+http://www.yourdomain.com)'
USER_AGENT = 'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'
DEFAULT_REQUEST_HEADERS = {
'Referer': 'http://www.weibo.com'
}
ITEM_PIPELINES = {
'weather.pipelines.WeatherPipeline': 1
}
DOWNLOAD_DELAY = 0.5# -*- coding: utf-8 -*-
import scrapy
from bs4 import BeautifulSoup
from weather.items import WeatherItem
class WeatherSpider(scrapy.Spider):
name = "weather_spider2"
allowed_domains = ["sina.com.cn"]
start_urls = ['http://weather.sina.com.cn']
def parse(self, response):
html_doc = response.body
#html_doc = html_doc.decode('utf-8')
soup = BeautifulSoup(html_doc)
itemTemp = {}
itemTemp['city'] = soup.find(id='slider_ct_name')
tenDay = soup.find(id='blk_fc_c0_scroll')
itemTemp['date'] = tenDay.findAll("p", {"class": 'wt_fc_c0_i_date'})
itemTemp['dayDesc'] = tenDay.findAll("img", {"class": 'icons0_wt'})
itemTemp['dayTemp'] = tenDay.findAll('p', {"class": 'wt_fc_c0_i_temp'})
item = WeatherItem()
for att in itemTemp:
item[att] = []
if att == 'city':
item[att] = itemTemp.get(att).text
continue
for obj in itemTemp.get(att):
if att == 'dayDesc':
item[att].append(obj['title'])
else:
item[att].append(obj.text)
return item
最后进入到weather文件夹内，开始运行scrapy。
可以先查看一下scrapy的命令有那些，在主文件夹内查看和在项目文件中查看是两个效果。
Scrapy 0.24.6 - project: weather Usage: scrapy <command> [options] [args] Available commands: bench Run quick benchmark test check Check spider contracts crawl Run a spider deploy Deploy project in Scrapyd target edit Edit spider fetch Fetch a URL using the Scrapy downloader genspider Generate new spider using pre-defined templates list List available spiders parse Parse URL (using its spider) and print the results runspider Run a self-contained spider (without creating a project) settings Get settings values shell Interactive scraping console startproject Create new project version Print Scrapy version view Open URL in browser, as seen by Scrapy Use "scrapy <command> -h" to see more info about a command
我们可以使用scrapy crawl weather_spider1或者scrapy crawl weather_spider2.然后在主文件夹内生成一个wea.txt的文件打开之后就是今天的天气。
city:北京 date: 05-11 day:多云(20°C ) night:多云( 11°C) date: 05-12 day:晴(27°C ) night:晴( 11°C) date: 05-13 day:多云(29°C ) night:晴( 17°C) date: 05-14 day:多云(29°C ) night:多云( 19°C) date: 05-15 day:晴(26°C ) night:晴( 12°C) date: 05-16 day:晴(27°C ) night:晴( 16°C) date: 05-17 day:阴(29°C ) night:晴( 19°C) date: 05-18 day:晴(29°C ) night:少云( 16°C) date: 05-19 day:局部多云(31°C ) night:少云( 16°C) date: 05-20 day:局部多云(29°C ) night:局部多云( 16°C)
标签：爬虫 python beautifulsoup xpath scrapy
原文地址：http://blog.csdn.net/djd1234567/article/details/45642375
