标签：segment include www. ast 处理 register 算法 alt 介绍
本文介绍 ElasticSearch启动时如何创建、加载Analyzer，主要的参考资料是Lucene中关于Analyzer官方文档介绍、ElasticSearch6.3.2源码中相关类：AnalysisModule、AnalysisPlugin、AnalyzerProvider、各种Tokenizer类和它们对应的TokenizerFactory。另外还参考了一个具体的基于ElasticSearch采用HanLP进行中文分词的插件：elasticsearch-analysis-hanlp
这篇文章的主要目的是搞懂：AnalysisModule、AnalysisPlugin、AnalyzerProvider、某个具体的Tokenizer，比如HanLPStandardAnalyzer、和TokenizerFactory 之间的关系。这里面肯定是用过了某个（某些）设置模式的。搞懂了这个自己也能照葫芦画瓢，开发自定义的Plugin了。
对比HanLP中文分词器和ElasticSearch中内置的标准分词器（StandardTokenizer），发现elasticsearch-analysis-hanlp的实现方法和ElasticSearch中实现的标准分词插件二者几乎是一个套路。
HanLP提供了各种各样的中文分词方式，比如：标准分词、索引分词、NLP分词……因此，HanLPTokenizerFactory实现了TokenizerFactory，实现了create()方法，负责创建各类分词器。
这种写法和ElasticSearch源码里面的StandardTokenizerFactory写法如出一辙。
把Analyzer想象成一部生产Token的机器，输入Text，输出Token。
An Analyzer builds TokenStreams, which analyze text. It thus represents a policy for extracting index terms from text.
这部机器可以以不同的方式生产Token。比如：对于英文，一般以文本中的空格作为分隔符，输入Text，输出Token。
对于中文，中文文本没有空格了，因此需要借助一些中文分词算法，输入Text，输出Token。
对于HTML这样的文本，那就需要根据HTML标签作为分隔符，输入Text，输出Token。
TokenStreamComponents内部类封装了生产Token的方式，看源码注释This class encapsulates the outer components of a token stream.It provides access to the source Tokenizer and .... 。主要是封装了Tokenizer
/**
* This class encapsulates the outer components of a token stream. It provides
* access to the source ({@link Tokenizer}) and the outer end (sink), an
* instance of {@link TokenFilter} which also serves as the
* {@link TokenStream} returned by
* {@link Analyzer#tokenStream(String, Reader)}.
*/
public static class TokenStreamComponents {
/**
* Original source of the tokens.
*/
protected final Tokenizer source;
/**
* Sink tokenstream, such as the outer tokenfilter decorating
* the chain. This can be the source if there are no filters.
*/
protected final TokenStream sink;若要自定义Analyzer，只需继承Analyzer类，重写createComponents()方法，提供一个Tokenizer就可以了。比如：HanLPStandardAnalyzer重写的方法如下：
@Override
protected Analyzer.TokenStreamComponents createComponents(String fieldName) {
// AccessController.doPrivileged((PrivilegedAction) () -> HanLP.Config.Normalization = true);
Tokenizer tokenizer = new HanLPTokenizer(HanLP.newSegment(), configuration);
return new Analyzer.TokenStreamComponents(tokenizer);
}另外，也可参考ElasticSearch中提供的StandardAnalyzer.java，它实现了ElasticSearch查询分析过程中的标准分词，它继承了StopwordAnalyzerBase.java，这样可以在生产Token的时候，过滤掉 stop words。
AnalyzerProvider封装了Analyzer，它的构造方法实例化一个Analyzer，并为Analyzer 提供了一些名称、版本相关的信息：
public class HanLPAnalyzerProvider extends AbstractIndexAnalyzerProvider<Analyzer> {
private final Analyzer analyzer;AbstractIndexAnalyzerProvider 里面有 name 和 Version信息(Constructs a new analyzer component, with the index name and its settings and the analyzer name.)
public abstract class AbstractIndexAnalyzerProvider<T extends Analyzer> extends AbstractIndexComponent implements AnalyzerProvider<T> {
private final String name;
protected final Version version;AnalysisHanLPPlugin负责注册各种各样的分词器。在定义索引的时候需要指定某个字段的Analyzer名称，比如下面 name 字段中的文本在都使用名称为hanlp_standard分词器分词后，写入ElasticSearch索引。
"name": {
"type": "text",
"analyzer": "hanlp_standard",
"fields": {
"raw": {
"type": "keyword"
}
}
},
AnalysisPlugin主要是下面三个方法，用来获取：CharFilter、TokenFilter、Tokenizer。关于这三个的区别可参考下节：索引分析过程。
/**
* Override to add additional {@link CharFilter}s. See {@link #requriesAnalysisSettings(AnalysisProvider)}
* how to on get the configuration from the index.
*/
default Map<String, AnalysisProvider<CharFilterFactory>> getCharFilters() {
return emptyMap();
}
/**
* Override to add additional {@link TokenFilter}s. See {@link #requriesAnalysisSettings(AnalysisProvider)}
* how to on get the configuration from the index.
*/
default Map<String, AnalysisProvider<TokenFilterFactory>> getTokenFilters() {
return emptyMap();
}
/**
* Override to add additional {@link Tokenizer}s. See {@link #requriesAnalysisSettings(AnalysisProvider)}
* how to on get the configuration from the index.
*/
default Map<String, AnalysisProvider<TokenizerFactory>> getTokenizers() {
return emptyMap();
}这里主要参考ElasticSearch启动过程中相关源代码。在创建PluginService过程中初始化各种Analyzer， Node.java
//加载 modules 和 plugins 目录下的内容
this.pluginsService = new PluginsService(tmpSettings, environment.configFile(), environment.modulesFile(), environment.pluginsFile(), classpathPlugins);貌似是通过创建的ClassLoader，不管是module还是plugin都视为bundle，以SPI方式接入底层Lucene，PluginService.java
// load modules
if (modulesDirectory != null) {
Set<Bundle> modules = getModuleBundles(modulesDirectory);
for (Bundle bundle : modules) {
modulesList.add(bundle.plugin);
}
seenBundles.addAll(modules);
}
// now, find all the ones that are in plugins/
if (pluginsDirectory != null) {
List<BundleCollection> plugins = findBundles(pluginsDirectory, "plugin");
for (final BundleCollection plugin : plugins) {
final Collection<Bundle> bundles = plugin.bundles();
for (final Bundle bundle : bundles) {
pluginsList.add(bundle.plugin);
}
seenBundles.addAll(bundles);
pluginsNames.add(plugin.name());
}加载 module/plugin jar文件：
try (DirectoryStream<Path> jarStream = Files.newDirectoryStream(dir, "*.jar")) {
for (Path jar : jarStream) {
// normalize with toRealPath to get symlinks out of our hair
URL url = jar.toRealPath().toUri().toURL();
if (urls.add(url) == false) {
throw new IllegalStateException("duplicate codebase: " + url);
}
}
}
//...
// create a child to load the plugin in this bundle
ClassLoader parentLoader = PluginLoaderIndirection.createLoader(getClass().getClassLoader(), extendedLoaders);
ClassLoader loader = URLClassLoader.newInstance(bundle.urls.toArray(new URL[0]), parentLoader);
当PluginService载入了所有的plugin后，过滤出与Analysis相关的Plugin，创建AnalysisModule
//从plugin service 中过滤出 与Analysis相关的plugin
AnalysisModule analysisModule = new AnalysisModule(this.environment, pluginsService.filterPlugins(AnalysisPlugin.class));注册各种分词器、filters、analyzer的名称：（这样在创建索引的时候，为某个索引字段指定分词器，就是用的这里的注册了的名称）
NamedRegistry<AnalysisProvider<CharFilterFactory>> charFilters = setupCharFilters(plugins);
NamedRegistry<AnalysisProvider<TokenFilterFactory>> tokenFilters = setupTokenFilters(plugins, hunspellService);
NamedRegistry<AnalysisProvider<TokenizerFactory>> tokenizers = setupTokenizers(plugins);
NamedRegistry<AnalysisProvider<AnalyzerProvider<?>>> analyzers = setupAnalyzers(plugins);
//....
private NamedRegistry<AnalysisProvider<AnalyzerProvider<?>>> setupAnalyzers(List<AnalysisPlugin> plugins) {
NamedRegistry<AnalysisProvider<AnalyzerProvider<?>>> analyzers = new NamedRegistry<>("analyzer");
analyzers.register("default", StandardAnalyzerProvider::new);
analyzers.register("standard", StandardAnalyzerProvider::new);
//....
public StandardAnalyzerProvider(IndexSettings indexSettings, Environment env, String name, Settings settings) {
//....
standardAnalyzer = new StandardAnalyzer(stopWords);
standardAnalyzer.setVersion(version);
}引用一段《An Introduction to Information Retrieval》中关于 token、type、term、dictionary概念的解释：(这里的type和ElasticSearch索引中的type是不一样的，ElasticSearch索引中的type以后版本将不支持了)
A token is an instance of a sequence of characters in some particular document that are grouped together as a useful semantic unit for processing. A type is the class of all tokens containing the same character sequence. A term is a (perhaps normalized) type that is included in the IR system‘s dictionary.
For example, if the document to be indexed is to
sleep perchance to dream, then there are 5 tokens, but only 4 types (since there are 2 instances of to). However, if to is omitted from the index (as a stop word) then there will be only 3 terms: sleep, perchance, and dream.
个人觉得Tokenization和Analysis过程有交叉的地方。Lucene中定义的Analysis是指：将字符串转化成Tokens的过程，Analysis主要有四个方面：
The analysis package provides the mechanism to convert Strings and Readers into tokens that can be indexed by Lucene. There are four main classes in the package from which all analysis processes are derived. These are:
这四个的区别如下：（以中文处理举例）
比如一句中文：“这是一篇关于ElasticSearch Analyzer的文章”，CharFilter过滤其中的某个字。Tokenizer是将这句话进行中文分词：这是、一篇、关于、ElasticSearch、Analyzer、的、文章；分词得到的结果就是一个个的Token。TokenFilter则是过滤某些Token。
The Analyzer is a factory for analysis chains. Analyzers don‘t process text, Analyzers construct CharFilters, Tokenizers, and/or TokenFilters that process text. An Analyzer has two tasks: to produce TokenStreams that accept a reader and produces tokens, and to wrap or otherwise pre-process Reader objects.
具体可参考：Lucene7.6.0。在Lucene中，Analyzer不处理文本，它只是构建CharFilters、Tokenizer、TokenFilters， 然后让它们来处理文本。
ElasticSearch6.3.2源码
HanLP进行中文分词的插件：elasticsearch-analysis-hanlp
原文：https://www.cnblogs.com/hapjin/p/10151887.html
ElasticSearch 启动时加载 Analyzer 源码分析
标签：segment include www. ast 处理 register 算法 alt 介绍
原文地址：https://www.cnblogs.com/hapjin/p/10151887.html
