码迷,mamicode.com
首页 > 编程语言 > 详细

【爬虫】如何用python+selenium网页爬虫

时间:2019-01-18 10:54:25      阅读:176      评论:0      收藏:0      [点我收藏+]

标签:expected   until   声明   www.   ops   ret   condition   ima   select   

一、前提

爬虫网页(只是演示,切勿频繁请求):https://www.kaola.com/

需要的知识:Python,selenium 库,PyQuery

参考网站:https://selenium-python-zh.readthedocs.io/en/latest/waits.html

 

二、简单的分析下网站

技术分享图片

 

三、步骤

  1.目标:

    1.open brower

    2.open url

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
import time
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from pyquery import PyQuery as py
brower = webdriver.Chrome() //定义一个brower ,声明webdriver,调用Chrome()方法
wait = WebDriverWait(brower,20) //设置一个全局等待时间
brower.get("https://www.kaola.com/")

  

  2.搜索【年货】

def search():
    try:
        brower.get("https://www.kaola.com/")
     //红包 close_windows = wait.until( EC.presence_of_element_located((By.XPATH,‘//div[@class="cntbox"]//div[@class="u-close"]‘)) )
     //输入框 input = wait.until( EC.presence_of_element_located((By.CSS_SELECTOR,‘#topSearchInput‘)) )
//搜索 submit = wait.until( EC.presence_of_element_located((By.XPATH,‘//*[@id="topSearchBtn"]‘)) ) close_windows.click() input.send_keys(‘年货‘) time.sleep(2) submit.click()
     //获取年货所有的页数 total = wait.until( EC.presence_of_element_located((By.CSS_SELECTOR,‘#resultwrap > div.splitPages > a:nth-child(11)‘)) ) return total.text except TimeoutException: return ‘error‘

  

  3.获取页面的信息

//使用pyQurey解析页面
def get_product(): wait.until( EC.presence_of_element_located((By.XPATH,‘//*[@id="result"]//li[@class="goods"]‘)) ) html = brower.page_source doc = py(html) goods = doc(‘#result .goods .goodswrap‘) for good in goods.items(): product = { ‘image‘ : good.find(‘a‘).attr(‘href‘), ‘title‘:good.find(‘a‘).attr(‘title‘), ‘price‘:good.find(‘.price .cur‘).text() } print(product)
def main():
  get_product()
  brower.close

  

.....后续更新

【爬虫】如何用python+selenium网页爬虫

标签:expected   until   声明   www.   ops   ret   condition   ima   select   

原文地址:https://www.cnblogs.com/totoro-cat/p/10286302.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!