码迷,mamicode.com
首页 > 编程语言 > 详细

python生成职业要求词云

时间:2017-08-10 13:41:19      阅读:165      评论:0      收藏:0      [点我收藏+]

标签:port   index   creat   logs   plt   jobs   img   image   dex   

接着上篇的说的,爬取了大数据相关的职位信息,http://www.17bigdata.com/jobs/。

# -*- coding: utf-8 -*-
"""
Created on Thu Aug 10 07:57:56 2017

@author: lenovo
"""

from wordcloud import WordCloud
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import jieba

def cloud(root,name,stopwords):
    filepath = root +‘\\‘ + name
    f = open(filepath,‘r‘,encoding=‘utf-8‘)
    txt = f.read()
    f.close()
    cut = jieba.cut(txt)
    words = []
    for i in cut:
        words.append(i)
    df = pd.DataFrame({‘words‘:words})
    s= df.groupby(df[‘words‘])[‘words‘].agg([(‘size‘,np.size)]).sort_values(by=‘size‘,ascending=False)
    s = s[~s.index.isin(stopwords[‘stopword‘])].to_dict()
    wordcloud = WordCloud(font_path =r‘E:\Python\machine learning\simhei.ttf‘,background_color=‘black‘)
    wordcloud.fit_words(s[‘size‘])
    plt.imshow(wordcloud)
    pngfile = root +‘\\‘ + name.split(‘.‘)[0] + ‘.png‘
    wordcloud.to_file(pngfile)
    
import os 
jieba.load_userdict(r‘E:\Python\machine learning\NLPstopwords.txt‘)
stopwords = pd.read_csv(r‘E:\Python\machine learning\StopwordsCN.txt‘,encoding=‘utf-8‘,index_col=False)
for root,dirs,file in os.walk(r‘E:\职位信息‘):
    for name in file:
        if name.split(‘.‘)[-1]==‘txt‘:
            print(name)
            cloud(root,name,stopwords)        

词云如图所示:

技术分享

可以看出有些噪声词没能被去除,比如相关、以上学历等无效词汇。本想通过DF判断停用词,但是我爬的时候没顾及到这个问题,外加本身记录数也不高,就没再找职位信息的停用词。当然也可看出算法和经验是很重要的。加油

python生成职业要求词云

标签:port   index   creat   logs   plt   jobs   img   image   dex   

原文地址:http://www.cnblogs.com/chenyaling/p/7338416.html

踩
(0)
赞
(0)
   
举报
评论 一句话评论(0)
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!