码迷，mamicode.com
首页 > 其他好文 > 详细

爬虫-BS4

时间：2020-06-25 19:49:45      阅读：40      评论：0      收藏：0      [点我收藏+]

标签：输入   css   div   port   处理   his   lang   不用   ring   

简介
BeautifulSoup提供一些简单的、python式的函数用来处理导航、搜索、修改分析树等功能。它是一个工具箱，通过解析文档为用户提供需要抓取的数据，因为简单，所以不需要多少代码就可以写出一个完整的应用程序。BeautifulSoup自动将输入文档转换为Unicode编码，输出文档转换为utf-8编码。你不需要考虑编码方式，除非文档没有指定一个编码方式，这时你仅仅需要说明一下原始编码方式就可以了。BeautifulSoup已成为和lxml、html6lib一样出色的python解释器，为用户灵活地提供不同的解析策略或强劲的速度

from bs4 import BeautifulSoup

html = """
<html><head><title>The Dormouse‘s story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html, ‘lxml‘)
print(soup.prettify())
print(soup.title.string)

<html>
 <head>
  <title>
   The Dormouse‘s story
  </title>
 </head>
 <body>
  <p class="title" name="dromouse">
   <b>
    The Dormouse‘s story
   </b>
  </p>
  <p class="story">
   Once upon a time there were three little sisters; and their names were
   <a class="sister" href="http://example.com/elsie" id="link1">
    <!-- Elsie -->
   </a>
   ,
   <a class="sister" href="http://example.com/lacie" id="link2">
    Lacie
   </a>
   and
   <a class="sister" href="http://example.com/tillie" id="link3">
    Tillie
   </a>
   ;
and they lived at the bottom of a well.
  </p>
  <p class="story">
   ...
  </p>
 </body>
</html>
The Dormouse‘s story

标签选择器

# 选择元素
soup.p # 标签加其内部的所有内容，且只获取第一个匹配的标签

type(soup.p) # bs4.element.Tag

bs4.element.Tag

soup.p

<p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>

data = []
for i in soup.find(‘body‘).children:
    data.append(i)

data

[‘\n‘,
 <p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>,
 ‘\n‘,
 <p class="story">Once upon a time there were three little sisters; and their names were
 <a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
 and they lived at the bottom of a well.</p>,
 ‘\n‘,
 <p class="story">...</p>,
 ‘\n‘]

print(soup.body.prettify())

<body>
 <p class="title" name="dromouse">
  <b>
   The Dormouse‘s story
  </b>
 </p>
 <p class="story">
  Once upon a time there were three little sisters; and their names were
  <a class="sister" href="http://example.com/elsie" id="link1">
   <!-- Elsie -->
  </a>
  ,
  <a class="sister" href="http://example.com/lacie" id="link2">
   Lacie
  </a>
  and
  <a class="sister" href="http://example.com/tillie" id="link3">
   Tillie
  </a>
  ;
and they lived at the bottom of a well.
 </p>
 <p class="story">
  ...
 </p>
</body>

# 提取信息
soup.p.string
# 获取名称
soup.p.name 
soup.p.parent.name
# 获取属性
print(soup.p.attrs) # 返回字典
print(soup.p.attrs[‘name‘])
print(soup.p[‘name‘])
# 获取内容
print(soup.p.string)

soup.title.string

type(soup.title.string)

type(soup.title.contents)

soup.head

soup.p

soup.p.string

"The Dormouse‘s story"

嵌套选择

# 每一返回结果是bs4.element.Tag
print(type(soup.p))
print(type(soup.p.b))

print(soup.p.string)
print(soup.p.b.string)

<class ‘bs4.element.Tag‘>
<class ‘bs4.element.Tag‘>
The Dormouse‘s story
The Dormouse‘s story

关联选择

# 选取到了一个节点元素之后，如果想要获取它的直接子节点可以调用contents属性
print(soup.body.contents)  # contents返回的是列表类型

[‘\n‘, <p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>, ‘\n‘, <p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>, ‘\n‘, <p class="story">...</p>, ‘\n‘]

print(soup.title.contents)

["The Dormouse‘s story"]

list_ = soup.body.contents
len(list_)

7

子节点和子孙节点

# print(soup.body.children)
for i,child in enumerate(soup.body.children):
    print(i,child)

0 

1 <p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
2 

3 <p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
4 

5 <p class="story">...</p>
6

?

print(soup.body.descendants) # 生成器,递归查询所有子孙节点
for i,childs in enumerate(soup.body.descendants):
    print(i,childs)

<generator object Tag.descendants at 0x000002435F075ED0>
0 

1 <p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
2 <b>The Dormouse‘s story</b>
3 The Dormouse‘s story
4 

5 <p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
6 Once upon a time there were three little sisters; and their names were

7 <a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>
8  Elsie 
9 ,

10 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
11 Lacie
12  and

13 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
14 Tillie
15 ;
and they lived at the bottom of a well.
16 

17 <p class="story">...</p>
18 ...
19

?

父节点和祖先节点

print(soup.a.parent) # 返回其直接父节点即其下内容
type(soup.a.parent) # bs4.element.Tag

<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>





bs4.element.Tag

print(soup.a.parents) # 
for i,des in enumerate(soup.a.parents):
    print(i,des)

<generator object Tag.descendants at 0x000002435F075ED0>
0 <p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
1 <body>
<p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
</body>
2 <html><head><title>The Dormouse‘s story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
</body></html>
3 <html><head><title>The Dormouse‘s story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
</body></html>

兄弟节点

html = """
<html>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            Hello
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> 
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, ‘lxml‘)
print(‘Next Sibling‘, soup.a.next_sibling)
print(‘Prev Sibling‘, soup.a.previous_sibling)
print(‘Next Siblings‘, list(enumerate(soup.a.next_siblings)))
print(‘Prev Siblings‘, list(enumerate(soup.a.previous_siblings)))

Next Sibling 
            Hello
            
Prev Sibling 
            Once upon a time there were three little sisters; and their names were
            
Next Siblings [(0, ‘\n            Hello\n            ‘), (1, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>), (2, ‘ \n            and\n            ‘), (3, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>), (4, ‘\n            and they lived at the bottom of a well.\n        ‘)]
Prev Siblings [(0, ‘\n            Once upon a time there were three little sisters; and their names were\n            ‘)]

方法选择器

# find_all()、find()等方法，我们可以调用方法然后传入相应等参数就可以灵活地进行查询了
find_all(name , attrs , recursive , text , **kwargs)
find_all，顾名思义，就是查询所有符合条件的元素，可以给它传入一些属性或文本来得到符合条件的元素
soup.find_all(True) # 返回所有标签

html=‘‘‘
<div class="panel">
<div class="panel-heading">
        <h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
</ul>
</div>
</div>
‘‘‘
soup = BeautifulSoup(html, ‘lxml‘)

name属性

print(soup.find_all(name=‘ul‘)) # 查询所有ul标签，返回list
print(‘#‘*50)
print(type(soup.find_all(name=‘ul‘)))
print(type(soup.find_all(name=‘ul‘)[0]))

[<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>, <ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>]
##################################################
<class ‘bs4.element.ResultSet‘>
<class ‘bs4.element.Tag‘>

for ul in soup.find_all(name=‘ul‘):
    print(ul.find_all(name=‘li‘)) # 结果返回列表类型，列表种的每个元素依然是Tagl类型
    for li in ul.find_all(name=‘li‘):
        print(li.string)

[]
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>]
Foo
Bar
Jay
[]
[<li class="element">Foo</li>, <li class="element">Bar</li>]
Foo
Bar

attrs属性

print(soup.find_all(attrs={‘class‘:‘element‘})) # 传入的是attrs参数，参数的类型是字典类型

print(soup.find_all(‘ul‘,attrs={‘id‘:‘list-1‘}))

[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>, <li class="element">Foo</li>, <li class="element">Bar</li>]

常用属性如id,class可以不用attrs来传递，而直接用id和class_

print(soup.find_all(class_=‘element‘))

print(soup.find_all(id=‘list-1‘))

[<li class="element">Foo</li>,
 <li class="element">Bar</li>,
 <li class="element">Jay</li>,
 <li class="element">Foo</li>,
 <li class="element">Bar</li>]

text属性

# text参数可以用来匹配节点的文本，传入的形式可以是字符串，可以是正则表达式
import re
html=‘‘‘
<div class="panel">
    <div class="panel-body">
        <a>Hello, this is a link</a>
        <a>Hello, this is a link, too</a>
    </div>
</div>
‘‘‘
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, ‘lxml‘)
print(soup.find_all(text=re.compile(‘link‘))) # 返回所有匹配正则表达式的节点文本组成的列表

[‘Hello, this is a link‘, ‘Hello, this is a link, too‘]

find(name,attrs,recursive,text,**kwargs)
find()方法返回的是单个元素，也就是第一个匹配的元素

html=‘‘‘
<div class="panel">
<div class="panel-heading">
        <h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
</ul>
</div>
</div>
‘‘‘

from bs4 import BeautifulSoup
soup = BeautifulSoup(html,‘lxml‘)

print(soup.find(neme=‘div‘)) # Return only the first child of this Tag matching the given
print(‘#‘*10)
print(soup.find(name=‘ul‘))
print(‘#‘*10)
print(soup.find(attrs={‘panel-heading‘}))
print(‘#‘*10)
print(soup.find(class_=‘element‘))

None
##########
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
##########
<div class="panel-heading">
<h4>Hello</h4>
</div>
##########
<li class="element">Foo</li>

soup.find_parent() # 返回直接父节点
soup.find_parents() # 返回所有祖先节点
soup.find_next_sibling()# 返回后面第一个兄弟节点
soup.find_next_siblings() # 返回后面所有兄弟节点
soup.find_next() # 返回第一个符合条件的节点
soup.find_all_next() # 返回节点后所有符合条件的节点

CSS选择器

使用CSS选择器，只需要调用select()方法，传入相应的CSS选择器即可

html=‘‘‘
<div class="panel">
<div class="panel-heading">
        <h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
</ul>
</div>
</div>
‘‘‘
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, ‘lxml‘)

print(soup.select(‘.panel .panel-heading‘)) # 符合CSS选择器的节点组成的列表

print(soup.select(‘ul li‘)) # 符合CSS选择器的节点组成的列表

print(soup.select(‘#list-2 .element‘)) # 符合CSS选择器的节点组成的列表
print(type(soup.select(‘ul‘)[0])) # print(type(soup.select(‘ul‘)[0]))

[<div class="panel-heading">
<h4>Hello</h4>
</div>]

嵌套选择

soup.select(‘ul‘)

[<ul class="list" id="list-1">
 <li class="element">Foo</li>
 <li class="element">Bar</li>
 <li class="element">Jay</li>
 </ul>, <ul class="list list-small" id="list-2">
 <li class="element">Foo</li>
 <li class="element">Bar</li>
 </ul>]

for ul in soup.select(‘ul‘):
    print(ul.select(‘li‘))

[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>]
[<li class="element">Foo</li>, <li class="element">Bar</li>]

# 获取属性
for ul in soup.select(‘ul‘):
    print(ul[‘id‘])
    print(ul.attrs[‘id‘])
    print(ul.get_text()) # 获取文本

list-1
list-1

Foo
Bar
Jay

list-2
list-2

Foo
Bar

?

print(soup.find_all(‘div‘)[1])
[out]
<div class="panel-heading">
<h4>Hello</h4>
</div>
soup.find_all(‘div‘)[1].get_text()
[out]
‘\nHello\n‘
print(soup.find_all(‘div‘)[1].h4.string)
[out]
Hello
print(soup.find_all(‘div‘)[1].string)
[out]
None

爬虫-BS4

标签：输入   css   div   port   处理   his   lang   不用   ring   

原文地址：https://www.cnblogs.com/aspiration/p/13192445.html
踩
(0)
赞
(0)
   
举报
评论 一句话评论（0）
登录后才能评论！
分享档案
更多>
2021年07月29日 (22)
2021年07月28日 (40)
2021年07月27日 (32)
2021年07月26日 (79)
2021年07月23日 (29)
2021年07月22日 (30)
2021年07月21日 (42)
2021年07月20日 (16)
2021年07月19日 (90)
2021年07月16日 (35)
周排行
mamicode.com排行更多图片
更多
友情链接
兰亭集智   国之画   百度统计   站长统计   阿里云   chrome插件   新版天听网
关于我们 - 联系我们 - 留言反馈
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码！