码迷,mamicode.com
首页 > 其他好文 > 详细

爬虫实现(hpricot)

时间:2015-03-11 17:01:38      阅读:160      评论:0      收藏:0      [点我收藏+]

标签:

1.基本代码

 在gemfile中加入gem "hpricot",bundler install之后,在application。rb中require "hpricot" require "open-uri". 

 

 1 pp "===========begin============="
 2 url = "http://www.xiaochuncnjp.com/search.php?mod=forum&searchid=552&orderby=lastpost&ascdesc=desc&searchsubmit=yes&kw=%E6%90%AC%E5%AE%B6"
 3 doc = Hpricot(open(url))
 4 # 获取返回页面的编码,使用了gem rchardet。
 5 cd = CharDet.detect(doc.to_s)
 6 pp encoding = cd["encoding"]
 8 # pp doc.search("ul/.pbw")  #获取返回页面ul标签下class为pbw的元素
 9 doc.search("ul/.pbw").each do |item|
10   # pp timeStr = item.inner_html
11   pp titleStr = item.search("h3/a").inner_html
12   pp urlStr = item.search("h3").inner_html.to_s.gsub(/href="/, ‘href="http://www.xiaochuncnjp.com/)
13   pp contentStr = item.search("p")[1].inner_html     
14 end
15 pp "************end***********"

 

2。当链接的协议为https时,报certificate verify failed error,无法通过认证的错误。

  https是安全协议,要通过验证可以add this ssl_verify option to the top of the file.来解决

FROM:

 module OpenURI
  Options = {
    :proxy => true,
    :progress_proc => true,
    :content_length_proc => true,
    :http_basic_authentication => true,
  }

 TO:

 module OpenURI
  Options = {
    :proxy => true,
    :progress_proc => true,
    :content_length_proc => true,
    :http_basic_authentication => true,
    :ssl_verify => true
  }

 Change the part where it enables verification

 FROM:

    if target.class == URI::HTTPS
      require ‘net/https‘
      http.use_ssl = true
      http.enable_post_connection_check = true
      http.verify_mode = OpenSSL::SSL::VERIFY_PEER
      store = OpenSSL::X509::Store.new
      store.set_default_paths
      http.cert_store = store
    end

 TO:
    if target.class == URI::HTTPS
      require ‘net/https‘
      http.use_ssl = true
      http.enable_post_connection_check = true
      if options[:ssl_verify] == false
        http.verify_mode = OpenSSL::SSL::VERIFY_NONE
      else
        http.verify_mode = OpenSSL::SSL::VERIFY_PEER
      end
      store = OpenSSL::X509::Store.new
      store.set_default_paths
      http.cert_store = store
    end

 run it like this:

 open("https://someurl", :ssl_verify => false) {|f|
  print f.read
 }

3.页面乱码

   由于网页的编码方式不同意,当你摘录信息的时候,很容易出现乱码。因此,你需要根据网页的编码方式转换编码。这个过程使用到了rchardet插件。

4.rchardet的使用

  在gemfile中加入gem "rchardet",bundler install之后,在application。rb中require "rchardet".

 

cd = CharDet.detect(some_data)
  encoding = cd[‘encoding‘]
  confidence = cd[‘confidence‘] # 0.0 <= confidence <= 1.0
  eg: CharDet.detect("\xA4\xCF")  #=>  {"encoding"=>"EUC-JP", "confidence"=>0.99}

  

爬虫实现(hpricot)

标签:

原文地址:http://www.cnblogs.com/hello-ruby/p/4329945.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!