码迷,mamicode.com
首页 > 其他好文 > 详细

爬虫实战--使用Selenium模拟浏览器抓取淘宝商品美食信息

时间:2018-09-28 14:39:16      阅读:192      评论:0      收藏:0      [点我收藏+]

标签:enc   att   ==   特色   杭州   exce   wait   active   dea   

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from pyquery import PyQuery as pq
import re


browser = webdriver.Chrome()
wait = WebDriverWait(browser,10)
def search():
    try:
        browser.get("https://www.taobao.com")
        input = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR,#q)))
        submit = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,#J_TSearchForm > div.search-button > button)))
        input.send_keys(美食)
        submit.click()
        total = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR,#mainsrp-pager > div > div > div > div.total)))
        get_products()
        return total.text
    except TimeoutException:
        return search()

def next_page(page_number):
    try:
        input = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, #mainsrp-pager > div > div > div > div.form > input)))
        submit = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, #mainsrp-pager > div > div > div > div.form > span.btn.J_Submit)))
        input.clear()
        input.send_keys(page_number)
        submit.click()
        wait.until(EC.text_to_be_present_in_element((By.CSS_SELECTOR,#mainsrp-pager > div > div > div > ul > li.item.active > span),str(page_number)))
        get_products()
    except TimeoutException:
        next_page(page_number)

def get_products():
    wait.until(EC.presence_of_element_located((By.CSS_SELECTOR,"#mainsrp-itemlist .items .item")))
    html = browser.page_source
    doc = pq(html)
    items = doc(#mainsrp-itemlist .items .item).items()
    for item in items:
        product = {
            image:item.find(.pic .img).attr(src),
            price:item.find(.price).text(),
            deal:item.find(.deal-cnt).text()[:-3],
            title:item.find(.title).text(),
            shop:item.find(.shop).text(),
            location:item.find(.location).text()
        }
        print(product)
def main():
    total = search()
    total = int(re.compile((\d+)).search(total).group(1))
    for i in range(2,total + 1):
        next_page(i)

if __name__ ==  "__main__":
    main()
技术分享图片
 1 {title: 坚果小吃零食组合装好吃的\n美食\n休闲食品孕妇成人款散装混合一箱, deal: 1910, image: //g-search1.alicdn.com/img/bao/uploaded/i4/imgextra/i4/123456239/TB2CqDhqgZC2uNjSZFnXXaxZpXa_!!0-saturn_solar.jpg_180x180.jpg, shop: 果木时光旗舰店, price: ¥\n59.80, location: 江苏 泰州}
 2 {title: 三只松鼠 肉松饼456g传统糕点早餐点心办公室\n美食, deal: 68153, image: //g-search2.alicdn.com/img/bao/uploaded/i4/i3/725677994/TB2RFQ8cZbpK1RjSZFyXXX_qFXa_!!725677994-0-sm.jpg_180x180.jpg, shop: 天猫超市, price: ¥\n26.90, location: 上海}
 3 {title: 轩妈家蛋黄酥6枚装 红豆味雪媚娘麻薯网红糕点点心\n美食\n早餐零食, deal: 110112, image: //g-search2.alicdn.com/img/bao/uploaded/i4/i4/3282604381/O1CN011iEXcnPKoV6ODRN_!!3282604381.jpg_180x180.jpg, shop: 轩妈食品旗舰店, price: ¥\n45.00, location: 广西 南宁}
 4 {title: 【百草味零食大礼包】休闲网红小吃一箱组合整箱超大混装\n美食\n批发, deal: 196990, image: //g-search3.alicdn.com/img/bao/uploaded/i4/i3/628189716/TB2w6WHdMHqK1RjSZFgXXa7JXXa_!!628189716-0-item_pic.jpg_180x180.jpg, shop: 百草味旗舰店, price: ¥\n58.90, location: 浙江 杭州}
 5 {title: 桂花鸭南京盐水鸭1000g正宗江苏特产中秋送礼\n美食\n板鸭咸水鸭熟食, deal: 6665, image: //g-search3.alicdn.com/img/bao/uploaded/i4/i2/2095703462/O1CN011bRddgG0RUsR8tR_!!2095703462.jpg_180x180.jpg, shop: 桂花鸭旗舰店, price: ¥\n52.00, location: 江苏 南京}
 6 {title: 良品铺子零食大礼包抖音网红小吃吃货一箱组合整箱混装\n美食\n批发, deal: 158982, image: //g-search3.alicdn.com/img/bao/uploaded/i4/i1/619123122/TB1ODM4ev5TBuNjSspcXXbnGFXa_!!0-item_pic.jpg_180x180.jpg, shop: 良品铺子旗舰店, price: ¥\n59.90, location: 湖北 武汉}
 7 {title: 轩妈家蛋黄酥2盒装 红豆雪媚娘麻薯糕点点心\n美食\n早餐网红零食, deal: 55683, image: //g-search2.alicdn.com/img/bao/uploaded/i4/i2/3282604381/O1CN011iEXcnzrOwKPzVy_!!3282604381.jpg_180x180.jpg, shop: 轩妈食品旗舰店, price: ¥\n89.00, location: 广西 南宁}
 8 {title: 轩妈家蛋黄酥55g*6枚2盒手工网红糕点\n美食\n早餐零食小吃雪媚娘麻薯, deal: 11107, image: //g-search3.alicdn.com/img/bao/uploaded/i4/i1/681185851/O1CN011t5nqIzARzgEZZX-681185851.jpg_180x180.jpg, shop: 猫诚旗舰店, price: ¥\n89.00, location: 广西 南宁}
 9 {title: 南京特产金陵夫子庙小吃零食传统糕点\n美食\n6盒大礼包送礼品袋, deal: 1897, image: //g.alicdn.com/s.gif, shop: 鹊菓食品旗舰店, price: ¥\n59.90, location: 江苏 南京}
10 {title: 其妙麻薯干吃汤圆整箱面包早餐糯米糍驴打滚糍粑网红小吃零食\n美食, deal: 101377, image: //g.alicdn.com/s.gif, shop: 其妙旗舰店, price: ¥\n19.90, location: 福建 厦门}
11 {title: 脆皮烤肉五花肉 云南特产\n美食\n真空熟食香辣猪肉干非红烧肉送蘸料, deal: 2268, image: //g.alicdn.com/s.gif, shop: 傣小妹, price: ¥\n27.80, location: 云南 西双版纳}
12 {title: 20枚玫瑰鲜花饼云南特产玫瑰花饼胡先生正宗糕点早餐零食\n美食\n1kg, deal: 18481, image: //g.alicdn.com/s.gif, shop: mrhu旗舰店, price: ¥\n39.90, location: 云南 昆明}
13 {title: 轩妈家蛋黄酥55g*6枚/盒手工雪媚娘麻薯糕点\n美食\n早餐网红零食小吃, deal: 24889, image: //g.alicdn.com/s.gif, shop: 猫诚旗舰店, price: ¥\n45.00, location: 广西 南宁}
14 {title: 切糕小吃新疆特产纯手工正宗传统散装坚果点心玛仁糖零食孕妇\n美食, deal: 199, image: //g.alicdn.com/s.gif, shop: yuxi钰玺, price: ¥\n39.90, location: 新疆 乌鲁木齐}
15 {title: 其妙麻薯干吃汤圆整箱面包早餐糯米糍驴打滚糍粑网红小吃零食\n美食, deal: 42446, image: //g.alicdn.com/s.gif, shop: 香客食品专营店, price: ¥\n19.80, location: 福建 厦门}
16 {title: 云南脆皮烤肉五花肉猪\n美食\n刘立民休闲食品熟食成人款即食小吃真空, deal: 1216, image: //g.alicdn.com/s.gif, shop: 小牛的烤肉, price: ¥\n30.00, location: 云南 西双版纳}
17 {title: 南萃坊流心蛋黄饼20个800克传统糕点办公室网红零食小吃\n美食\n整箱, deal: 9557, image: //g.alicdn.com/s.gif, shop: 南萃坊旗舰店, price: ¥\n32.80, location: 浙江 杭州}
18 {title: 梅菜扣肉饼黄山烧饼梅干菜手工薄脆饼好吃的\n美食\n网红零食小吃16个, deal: 501, image: //g.alicdn.com/s.gif, shop: tb361290_55, price: ¥\n29.80, location: 安徽 黄山}
19 {title: 美食\n作家王刚自贡特色冷吃牛肉 105克-205克, deal: 10353, image: //g.alicdn.com/s.gif, shop: wg王刚1590, price: ¥\n49.90, location: 四川 自贡}
20 {title: 鲜花饼云南特产\n美食\n手工玫瑰饼五味糕点心零食1000g中秋礼盒送礼, deal: 4162, image: //g.alicdn.com/s.gif, shop: 花香随鲜花饼店, price: ¥\n34.80, location: 云南 昆明}
21 {title: 北方无糖零食曲阳缸炉烧饼特产正宗河北\n美食\n小吃芝麻饼传统纯手工, deal: 346, image: //g.alicdn.com/s.gif, shop: 付太安, price: ¥\n32.50, location: 河北 保定}
22 {title: 新品【三只松鼠_黄金肉松饼1.25kg/整箱】早餐面包食品糕点心\n美食, deal: 10148, image: //g.alicdn.com/s.gif, shop: 三只松鼠旗舰店, price: ¥\n42.90, location: 安徽 芜湖}
23 {title: 爱福龙须酥正宗手工糕点小吃点心龙须糖\n美食\n好吃的怀旧零食特产, deal: 239, image: //g.alicdn.com/s.gif, shop: 海艳食品专营店, price: ¥\n29.90, location: 湖北 随州}
打印后的结果为:

 

爬虫实战--使用Selenium模拟浏览器抓取淘宝商品美食信息

标签:enc   att   ==   特色   杭州   exce   wait   active   dea   

原文地址:https://www.cnblogs.com/zhuifeng-mayi/p/9718171.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!