码迷,mamicode.com
首页 > 其他好文 > 详细

NLP Calculate the similarity of any two articles resume version

时间:2020-01-31 21:16:24      阅读:140      评论:0      收藏:0      [点我收藏+]

标签:log   stop   alt   超级   cors   文章   再计算   语料库   ext   

 

技术图片

 

https://radimrehurek.com/gensim/auto_examples/index.html#core-tutorials


 

Calculate the similarity of any two course

-Design a program to implement the retrieval of similar courses on corsera by using NLP

-Implement input a coursara course to find the top 10 courses similar to it by using python and accuracy can be above 90% 

-Use the Topic model to solve this problem, mapping the text content to the Topic‘s dimensions, and then calculating its similarity. 

-Utilized :Python,NLTK,genius,Topic model,lsi,tf*idf


 

 

 

 

基于 语料库当年为了保研究生 向tsinghua university apply the master degree 

 

就是将两个"article"的文本内容映射到topic的维度,然后再计算其相似度。

技术图片

 

 

download: https://www.aminer.cn/citation

 

Calculate the similarity of any two articles

基于关键词 :https://www.jiqizhixin.com/articles/2018-11-14-17

有很多种方法可以提取出关键词

tf-idf算出词频 然后作为文本的向量

 

 

 

基于全量的英文维基百科(400多万文章,压缩后9个多G的语料)

基于LSI模型的课程索引建立完毕,我们以Andrew Ng教授的机器学习公开课为例,这门课程在我们的coursera_corpus文件的第211行,也就是:

>>> print courses_name[210]
Machine Learning

现在我们就可以通过lsi模型将这门课程映射到10个topic主题模型空间上,然后和其他课程计算相似度:
>>> ml_course = texts[210]
>>> ml_bow = dicionary.doc2bow(ml_course)
>>> ml_lsi = lsi[ml_bow]
>>> print ml_lsi
[(0, 8.3270084238788673), (1, 0.91295652151975082), (2, -0.28296075112669405), (3, 0.0011599008827843801), (4, -4.1820134980024255), (5, -0.37889856481054851), (6, 2.0446999575052125), (7, 2.3297944485200031), (8, -0.32875594265388536), (9, -0.30389668455507612)]
>>> sims = index[ml_lsi]
>>> sort_sims = sorted(enumerate(sims), key=lambda item: -item[1])

取按相似度排序的前10门课程:
>>> print sort_sims[0:10]
[(210, 1.0), (174, 0.97812241), (238, 0.96428639), (203, 0.96283489), (63, 0.9605484), (189, 0.95390636), (141, 0.94975704), (184, 0.94269753), (111, 0.93654782), (236, 0.93601125)]

第一门课程是它自己:
>>> print courses_name[210]
Machine Learning

第二门课是Coursera上另一位大牛Pedro Domingos机器学习公开课
>>> print courses_name[174]
Machine Learning

第三门课是Coursera的另一位创始人,同样是大牛的Daphne Koller教授的概率图模型公开课
>>> print courses_name[238]
Probabilistic Graphical Models

第四门课是另一位超级大牛Geoffrey Hinton的神经网络公开课,有同学评价是Deep Learning的必修课。
>>> print courses_name[203]
Neural Networks for Machine Learning

输入到住lsi主题模型中

# -*- coding:gb2312 -*-
from gensim import corpora, models, similarities
from nltk.tokenize import word_tokenize
from nltk.corpus import brown




//文本做一些格式上的处理 courses=[] temp="" for line in file(‘aaa‘): if(line!="\n"): temp =temp+line.strip()+"\t" else: courses.append(temp) temp="" courses_name = [] for course in courses: x=course.strip().split(‘\t‘) courses_name.append(x[0].strip(‘#*‘)) print courses_name[0:3] document=[‘#*AD‘,‘ADdd‘] document=document[0].decode(‘utf-8‘).lower() print document
//将文本分词 texts_tokenized = [[word.lower() for word in word_tokenize(document.decode(‘utf-8‘))] for document in courses] print texts_tokenized[0]
//去掉停用词 from nltk.corpus import stopwords english_stopwords = stopwords.words(‘english‘) texts_filtered_stopwords = [[word for word in document if not word in english_stopwords] for document in texts_tokenized] print texts_filtered_stopwords[0] english_punctuations = [‘,‘, ‘.‘, ‘:‘, ‘;‘, ‘?‘, ‘(‘, ‘)‘, ‘[‘, ‘]‘, ‘&‘, ‘!‘, ‘*‘, ‘@‘, ‘#‘, ‘$‘, ‘%‘] //词根
from nltk.stem.lancaster import LancasterStemmer texts_filtered = [[word for word in document if not word in english_punctuations] for document in texts_filtered_stopwords] print texts_filtered[0] st = LancasterStemmer() texts_stemmed = [[st.stem(word) for word in docment] for docment in texts_filtered] print texts_stemmed[0] all_stems = sum(texts_stemmed, []) stems_once = set(stem for stem in set(all_stems) if all_stems.count(stem) == 2)#去掉次数为2的低频词汇 texts = [[stem for stem in text if stem not in stems_once] for text in texts_stemmed] print texts f
rom gensim import corpora, models, similarities import logging logging.basicConfig(format=‘%(asctime)s : %(levelname)s : %(message)s‘, level=logging.INFO) // 构造词典 dictionary = corpora.Dictionary(texts) corpus = [dictionary.doc2bow(text) for text in texts]
//计算tf-idf tfidf = models.TfidfModel(corpus)
corpus_tfidf = tfidf[corpus]‘
lsi = models.LsiModel(corpus_tfidf, id2word=dictionary, num_topics=10)
//做成索引 index = similarities.MatrixSimilarity(lsi[corpus])
//输入第16门课
ml_course = texts[15]
//转成向量 ml_bow = dictionary.doc2bow(ml_course)
现在我们就可以通过lsi模型将这门课程映射到10个topic主题模型空间上,然后和其他课程计算相似 ml_lsi = lsi[ml_bow] print ml_lsi sims = index[ml_lsi] sort_sims = sorted(enumerate(sims), key=lambda item: -item[1]) print sort_sims[1:11] #print courses_name[23

  

 

NLP Calculate the similarity of any two articles resume version

标签:log   stop   alt   超级   cors   文章   再计算   语料库   ext   

原文地址:https://www.cnblogs.com/ldphoebe/p/12246493.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!