简单的网络爬虫是对一个url进行请求，并等待其返回响应。在数据量小的情况下很实用，但是当你的数据量很大，显然分布式爬虫就更占优势！关于分布式，一般是使用一台主机（master）充当多个爬虫的共享redis队列，其他主机（slave）采用远程连接master，关于redis如何安装，这里不多做介绍！
以爬虫伯乐在线的python文章为例，我的分布式爬虫由main01 main02 main03三个python文件构成，main01的主要任务是运行在master上，将文章的url爬取下来存入redis以供main02和main03读取解析数据。main01的主要代码如下：
# -*- coding: utf-8 -*-
from bs4 import BeautifulSoup
import requests
from redis import Redis
from lxml import etree
r = Redis.from_url("redis://x.x.x.x:6379", decode_responses=True)
def get_urls(url="http://python.jobbole.com/all-posts/"):
result = requests.get(url=url)
selector = etree.HTML(result.text)
links = selector.xpath(r‘//*[@id="archive"]/div/div[2]/p[1]/a[1]/@href‘)
for link in links:
r.sadd("first_urls", link)
next_url = extract_next_url(result.text)
if next_url:
get_urls(next_url)
def extract_next_url(html):
soup = BeautifulSoup(html, "lxml")
next_url = soup.select(‘a[class="next page-numbers"]‘)
for url in next_url:
url = str(url)
soup = BeautifulSoup(url, "lxml")
next_url = soup.a["href"]
return next_url
if __name__ == ‘__main__‘:
get_urls()
从本地连接master的redis可以看到，数据已经成功写入redis
下面是main02的代码：
# -*- coding: utf-8 -*-
import json
import codecs
import requests
from redis import Redis
from lxml import etree
from settings import *
import MySQLdb
r = Redis.from_url(url=REDIS_URL, decode_responses=True)
def parse_urls():
if "first_urls" in r.keys():
while True:
try:
url = r.spop("first_urls")
result = requests.get(url=url, timeout=10)
selector = etree.HTML(result.text)
title = selector.xpath(r‘//*[@class="entry-header"]/h1/text()‘)
title = title[0] if title is not None else None
author = selector.xpath(r‘//*[@class="copyright-area"]/a/text()‘)
author = author[0] if author is not None else None
items = dict(title=title, author=author, url=url)
insert_mysql(items)
except:
if "first_urls" not in r.keys():
print("爬取结束，关闭爬虫！")
break
else:
print("{}请求发送失败！".format(url))
continue
else:
parse_urls()
def insert_json(value):
file = codecs.open("save.json", "a", encoding="utf-8")
line = json.dumps(value, ensure_ascii=False) + "," + "\n"
file.write(line)
file.close()
def insert_mysql(value):
conn = MySQLdb.connect(MYSQL_HOST, MYSQL_USER, MYSQL_PASSWORD, MYSQL_DBNAME, charset="utf8", use_unicode=True)
cursor = conn.cursor()
insert_sql = ‘‘‘
insert into article(title, author, url) VALUES (%s, %s, %s)
‘‘‘
cursor.execute(insert_sql, (value["title"], value["author"], value["url"]))
conn.commit()
if __name__ == ‘__main__‘:
parse_urls()
main02和main03（同main02的代码可以一样）可以运行在你的本地机器，在main02中我们会先判断master的redis中是否已经生成url，如果没有main02爬虫会等待，直到master的redis存在url，才会进行下面的解析！
运行main02可以发现本地的mysql数据库中已经成功被写入数据！
以上就是一个简单的分布式爬虫，当然真正运行的时候，肯定是几个爬虫同时运行的，这里只是为了调试才先运行了main01！
