标签：content 数据结构 port etc test utf-8 orm 正文 form
1. 将新闻的正文内容保存到文本文件。
f=open(‘test.txt‘,‘w‘,encoding=‘utf-8‘) f.write(content) f.close()
2. 将新闻数据结构化为字典的列表:
import requests from bs4 import BeautifulSoup from datetime import datetime import re import pandas #获取新闻点击次数 def getClickCount(newsUrl): newsId = re.findall(‘\_(.*).html‘, newsUrl)[0].split(‘/‘)[1] clickUrl = ‘http://oa.gzcc.cn/api.php?op=count&id={}&modelid=80‘.format(newsId) clickStr = requests.get(clickUrl).text count = re.search("hits‘\).html\(‘(.*)‘\);",clickStr).group(1) return count def Writecontent(content) f=open(‘test.txt‘,‘w‘,encoding=‘utf-8‘) f.write(content) f.close() # 获取新闻的信息 def getNewDetail(url): resd = requests.get(url) resd.encoding = ‘utf-8‘ soupd = BeautifulSoup(resd.text, ‘html.parser‘) news={} news[‘title‘] = soupd.select(‘.show-title‘)[0].text info = soupd.select(‘.show-info‘)[0].text news[‘time‘] = info.lstrip(‘发布时间:‘)[0:19] dt = datetime.strptime(time, ‘%Y-%m-%d %H:%M:%S‘) if info.find(‘来源：‘) > 0: news[‘source‘] = info[info.find(‘来源：‘):].split()[0].lstrip(‘来源：‘) else: news[‘source‘] = ‘none‘ if info.find(‘作者：‘) > 0: news[‘author‘] = info[info.find(‘作者：‘):].split()[0].lstrip(‘作者：‘) else: news[‘author‘] = ‘none‘ news[‘clickcount‘]=getClickCount(url); content=soupd.select(‘.show-content‘)[0].text.strip() writeNewsContent(content) return news def getListPage(listPageUrl): res = requests.get(listPageUrl) res.encoding = ‘utf-8‘ soup = BeautifulSoup(res.text, ‘html.parser‘) newsList=[] for news in soup.select(‘li‘): if len(news.select(‘.news-list-title‘)) > 0: # 获取新闻模块链接 a = news.a.attrs[‘href‘] newsList.append(getNewsDetail(a)) return newsList url=‘http://news.gzcc.cn/html/xiaoyuanxinwen/‘ resn = requests.get(url) resn.encoding = ‘utf-8‘ soupn = BeautifulSoup(resn.text,‘html.parser‘) n = int(soupn.select(‘.a1‘)[0].text.rstrip(‘条‘)) for i in range(n,n+1): pageUrl = ‘http://news.gzcc.cn/html/xiaoyuanxinwen/{}.html‘.format(i) getListPage(pageUrl) newsTotal.extend(getListPage(pageUrl)) df=pandas.DataFrame(newsTotal) df.to_excel(‘test.xlsx‘)
3. 安装pandas，用pandas.DataFrame(newstotal)，创建一个DataFrame对象df.
df=pandas.DataFrame(newsTotal)
4. 通过df将提取的数据保存到csv或excel 文件。
df.to_excel(‘test.xlsx‘)
5. 用pandas提供的函数和方法进行数据分析：
print(df[[‘clickcount‘, ‘title‘, ‘source‘]].head(6))
print(df[(df[‘clickcount‘] > 3000) & (df[‘source‘] == ‘学校综合办‘)])
soulist = [‘国际学院‘, ‘学生工作处‘] print(df[df[‘source‘].isin(soulist)])
标签：content 数据结构 port etc test utf-8 orm 正文 form
原文地址：https://www.cnblogs.com/abcdcd/p/8810370.html
