码迷,mamicode.com
首页 > 其他好文 > 详细

遇到Io阻塞时会切换任务之【爬虫版】

时间:2019-01-24 22:54:45      阅读:128      评论:0      收藏:0      [点我收藏+]

标签:com   tar   urllib   data   操作   nal   imp   receive   event   

 

 

#! /usr/bin/env python3
# -*- coding:utf-8 -*-

from urllib import request
import gevent,time
from gevent import monkey

monkey.patch_all()    # 把当前程序中的所有io操作都做上标记

def spider(url):
    print("GET:%s" % url)
    resp = request.urlopen(url)
    data = resp.read()
    print("%s bytes received from %s.." % (len(data), url))

urls = [
    "https://weibo.com/",
    "https://www.baidu.com/",
    "https://www.yinxiang.com/"
]

start_time = time.time()
for url in urls:
    spider(url)
print("同步耗时:",time.time() - start_time)

async_time_start = time.time()
gevent.joinall([
    gevent.spawn(spider,"https://weibo.com/"),
    gevent.spawn(spider,"https://www.baidu.com/"),
    gevent.spawn(spider,"https://www.yinxiang.com/"),
])
print("异步耗时:",time.time() - async_time_start)

输出

GET:https://weibo.com/
6117 bytes received from https://weibo.com/..
GET:https://www.baidu.com/
227 bytes received from https://www.baidu.com/..
GET:https://www.yinxiang.com/
16505 bytes received from https://www.yinxiang.com/..
同步耗时: 0.23460006713867188
GET:https://weibo.com/
GET:https://www.baidu.com/
GET:https://www.yinxiang.com/
227 bytes received from https://www.baidu.com/..
16505 bytes received from https://www.yinxiang.com/..
6117 bytes received from https://weibo.com/..
异步耗时: 0.0946660041809082

 

遇到Io阻塞时会切换任务之【爬虫版】

标签:com   tar   urllib   data   操作   nal   imp   receive   event   

原文地址:https://www.cnblogs.com/sea-stream/p/10317051.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!