码迷,mamicode.com
首页 > 其他好文 > 详细

完整的中英文词频统计

时间:2018-10-18 13:17:04      阅读:165      评论:0      收藏:0      [点我收藏+]

标签:reverse   png   str   read   else   The   中英文   com   for   

#读取字符串str
f = open(zz.txt,r,encoding=utf-8)
strbig= f.read()
f.close()

sep =‘‘‘.,;:?!-_‘‘‘
for ch in sep:
    strbig = strbig.replace(ch, )

strbig = strbig.lower()
print(strbig)

strlist = strbig.split()
print(len(strlist),strlist)

strset = set(strlist)
exclude = {a,the,and,i,you}
strset = strset-exclude
print(len(strset),strset)

strdict ={}
for word in strset:
    strdict[word] = strlist.count(word)
print(len(strdict),strdict)

ccList = list(strdict.items())
print(ccList)
ccList.sort(key=lambda x: x[1], reverse=True)
print(ccList)

for i in range(20):
    b = ccList[i]
    print(b)

技术分享图片

 


 

fo = open(‘a.txt‘,‘r‘,encoding=‘utf-8‘)
shz=fo.read()
fo.close()
print(shz)
 
# jieba3种模式分词
import jieba
print(list(jieba.cut(shz)))
print(list(jieba.cut(shz,cut_all=True)))
print(list(jieba.cut_for_search(shz)))
 
 
s1=‘‘‘, 。 ; : - ! ? 、 “ ”‘‘‘    #标点符号转换成空格
for ch in s1:
    shz=shz.replace(ch,‘ ‘)
    print(shz)
 
strlist=shz.split()
print(len(strlist),strlist)   #分解提取词语
 
 
wordsls=jieba.lcut(shz)
wcdict={}
for word in wordsls:   #词为1的删除
    if len(word)==1:
        continue
    else:
        wcdict[word]=wcdict.get(word,0)+1
print(wordsls)
 
# # 按词频排序
shls=list(wcdict.items())
shls.sort(key=lambda x:x[1],reverse=True)
print(shls)
#
# # 输出TOP15
for i in range(15):
    print(shls[i])

 技术分享图片

 

完整的中英文词频统计

标签:reverse   png   str   read   else   The   中英文   com   for   

原文地址:https://www.cnblogs.com/xsxsx/p/9809753.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!