码迷,mamicode.com
首页 > 编程语言 > 详细

多线程下载

时间:2019-10-05 22:54:45      阅读:166      评论:0      收藏:0      [点我收藏+]

标签:win   map   函数   htm   gecko   视频   targe   open   proc   

多线程下载

梨视频下载为例

要点:

1.获取所有视频的url,定义下载函数getdata

2.将下载数据保存到列表,定义数据保存函数savefunc

3.from multiprocessing.dummy import Pool

? pool = Pool(4)

? pool.get(getdata,url)

? pool.map(savefunc,datalist)

import requests,re
from multiprocessing.dummy import Pool
from lxml import etree
from random import randint

target_url = 'https://www.pearvideo.com'
headers = {
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36',
    }
response = requests.get(url = target_url,headers = headers)
ht = response.text
tree = etree.HTML(ht)
link_list = tree.xpath("//*[@id = 'actRecommendCont'][1]//a[@class = 'actcont-detail actplay']/@href")
videoUrl = []
for link in link_list:
    detail = 'https://www.pearvideo.com/' + link
    response1 = requests.get(url = detail,headers=headers)
    ht = response1.text
    reg = 'var contId.*?srcUrl="(.*?)"'
    link = re.findall(reg,ht,re.S)[0]
    videoUrl.append(link)
    
def getVideoDate(url):
    return requests.get(url = url,headers = headers).content
pool = Pool(4)
videoData_list = pool.map(getVideoDate,videoUrl)

def saveVideo(VD):
    fn = int(randint(1,999))
    with open('./%s.mp4' % fn,'wb') as f:
        f.write(VD)
        print('视频%s下载完毕' % fn)
pool.map(saveVideo,videoData_list)

多线程下载

标签:win   map   函数   htm   gecko   视频   targe   open   proc   

原文地址:https://www.cnblogs.com/notfind/p/11625879.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!