标签:amp pre rdma lda 位置 ext pac ast util
标记出一句话中所有关键词
inpu:我想买苹果手机,请问哪里可以买苹果手机
keyword:"苹果", "苹果手机", "哪里"
result:我想买[[苹果手机]],请问[[哪里]]可以买[[苹果手机]]
10w条耗时:41ms
难点:需要考虑单词重叠问题(overlap),例如“苹果手机”同时包含两个关键词,只标记一次。
pom文件引入hanlp,使用其中ac算法
<dependency>
<groupId>com.hankcs</groupId>
<artifactId>hanlp</artifactId>
<version>portable-1.7.8</version>
</dependency>
package com.bincoder.StringUtils;
import com.hankcs.hanlp.collection.AhoCorasick.AhoCorasickDoubleArrayTrie;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.TreeMap;
public class KeywordMatch {
/**
* 构建ac自动机
*/
public static AhoCorasickDoubleArrayTrie<String> buildAcdt(List<String> keywords){
AhoCorasickDoubleArrayTrie<String> acdt = new AhoCorasickDoubleArrayTrie<>();
TreeMap<String, String> map = new TreeMap<>();
for(String keyword : keywords){
map.put(keyword, keyword);
}
acdt.build(map);
return acdt;
}
public static String highLight(String originText, AhoCorasickDoubleArrayTrie<String> acdt){
List<int[]> hitLocationList = new ArrayList<>();
// ac算法匹配关键词
acdt.parseText(originText, (begin, end, value)->{
int[] indexPair = new int[2];
indexPair[0] = begin;
indexPair[1] = end-1;
hitLocationList.add(indexPair);
});
// 构建bitmap
byte[] posStatus = new byte[originText.length()];
for(int[] item : hitLocationList){
for(int i=item[0]; i<item[1]; i++){
posStatus[i] = 1;
}
}
// 字符串拼接
int lastStatus = 0;
char[] charArray = originText.toCharArray();
StringBuilder stringBuilder = new StringBuilder();
for(int i=0; i<posStatus.length; i++){
if(posStatus[i] == lastStatus){
stringBuilder.append(charArray[i]);
}else if(0 == lastStatus){
stringBuilder.append("[[").append(charArray[i]);
lastStatus = 1;
}else if(1 == lastStatus){
stringBuilder.append(charArray[i]).append("]]");
lastStatus = 0;
}
if(i == posStatus.length-1 && 1 == lastStatus){
stringBuilder.append("]]");
}
}
return stringBuilder.toString();
}
public static void main(String[] args) {
String text = "我想买苹果手机,请问哪里可以买苹果手机";
List<String> keywords = Arrays.asList("苹果", "苹果手机", "哪里");
AhoCorasickDoubleArrayTrie<String> acdt = KeywordMatch.buildAcdt(keywords);
String result = KeywordMatch.highLight(text, acdt);
System.out.println("inpu:" + text);
System.out.println("result:" + result);
long start = System.currentTimeMillis();
for(int i=0; i<100000; i++){
result = KeywordMatch.highLight(text, acdt);
}
long end = System.currentTimeMillis();
long total = end - start;
System.out.println("耗时:" + total + "ms");
}
}
标签:amp pre rdma lda 位置 ext pac ast util
原文地址:https://www.cnblogs.com/bincoding/p/13613492.html