Elasticsearch 教程（九）： 中文分词 – 结巴分词

上一章节中我们牛刀小试了一下Elasticsearch，使用Elasticsearch的标准分词器对我们的英文问候语进行了分词，但是，这个分词器对中文的分词结果却不尽人意
你也知道，我们中文没有所谓的空格分隔的词，只有黏在一起的字组成的词
于是，我们必须装一个中文分词插件
百度或 Google 一下Elasticsearch 中文分词，你会发现大量的文章和大量的分词器
但这里，我们只使用 结巴分词

结巴分词

结巴分词 是一个简单的相当流行的Python中文分词组件
它有以下特点
1、支持三种分词模式：
1、精确模式，试图将句子最精确地切开，适合文本分析；
2、全模式，把句子中所有的可以成词的词语都扫描出来, 速度非常快，但是不能解决歧义；
3、搜索引擎模式，在精确模式的基础上，对长词再次切分，提高召回率，适合用于搜索引擎分词。
2、支持繁体分词
3、支持自定义词典
4、MIT授权协议
虽然它的官方版本使用Python语言开发
但很多热心的开发者把它翻译成了各种语言，其中Github上的Cheng Zhang用户就制作了一个Elasticsearch版本的结巴分词插件

Elasticsearch 结巴分词插件

Elasticsearch结巴分词插件的官方地址为 https://github.com/sing1ee/elasticsearch-jieba-plugin 你可以在简介中看到各个版本的结巴分词插件 因为我们使用的是6.3.0的版本，所以我们选择下载v6.0.1

安装

下载完成后先不着急解压，正如官方文档所言，需要使用 gradle 工具来生成jar文件
所以我们首先要做的就是安装Gradle，安装过程我们就不详细介绍了，网上搜索一大堆
1、打开一个新的终端，使用cd命令跳转到elasticsearch-jieba-plugin-6.0.1.zip所在的目录
2、使用下面的命令对elasticsearch-jieba-plugin-6.0.1.zip进行解压

unzip elasticsearch-jieba-plugin-6.0.1.zip

如果你的电脑是Windows，则直接右键点击解压即可
3、使用cd elasticsearch-jieba-plugin-6.0.1命令跳转到elasticsearch-jieba-plugin-6.0.1目录
cd elasticsearch-jieba-plugin-6.0.1
4、接着运行gradle pz进行打包

Starting a Gradle Daemon (subsequent builds will be faster) Download http://maven.aliyun.com/nexus/content/groups/public/org/elasticsearch/elasticsearch/6.0.0/elasticsearch-6.0.0.pom Download http://maven.aliyun.com/nexus/content/groups/public/org/apache/logging/log4j/log4j-api/2.7/log4j-api-2.7.pom .... Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0. See https://docs.gradle.org/4.6/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 3m 54s 4 actionable tasks: 4 executed

命令执行的过程时间有点长，如果不发生任何意外，会提示构建成功

BUILD SUCCESSFUL in 3m 54s 4 actionable tasks: 4 executed

5、接下里来使用cp命令将build/distributions/目录下的elasticsearch-jieba-plugin-6.0.0.zip压缩包拷贝到Elasticsearch安装目录下的plugins目录下

cp build/distributions/elasticsearch-jieba-plugin-6.0.0.zip /usr/local/elasticsearch/6.3.0/plugins

如果你是根据我们之前教程的安装目录，那么
1、在 Unix 系统中目录应该是/usr/local/elasticsearch/6.3.0/plugins2、Windows 系统中目录应该是d:\devops\elasticsearch-6.3.06、使用cd命令跳转到Elasticsearch安装目录下的plugins目录下

cd /usr/local/elasticsearch/6.3.0/plugins

7、然后运行下面的命令来安装我们的结巴分词插件

unzip elasticsearch-jieba-plugin-6.0.0.zip

运行结果如下

Archive: elasticsearch-jieba-plugin-6.0.0.zip creating: jieba/ inflating: jieba/jieba-analysis-1.0.2.jar inflating: jieba/elasticsearch-jieba-plugin-6.0.0.jar inflating: jieba/plugin.xml        
  inflating: jieba/plugin-descriptor.properties  
   creating: jieba/dic/ inflating: jieba/dic/sougou.dict   
  inflating: jieba/dic/user.dict     
  inflating: jieba/dic/stopwords.txt

8、然后运行下面的命令来删除elasticsearch-jieba-plugin-6.0.0.zip

rm elasticsearch-jieba-plugin-6.0.0.zip

9、重新启动Elasticsearch

重新启动 Elasticsearch 失败

因为当前的版本是 6.3.0 ，而结巴插件的版本是 6.0.0，所以启动过程必然报错
像我我连续报了两个错误

2018-06-28T18:14:19,360][INFO ][o.e.n.Node ] [] initializing ... [2018-06-28T18:14:19,463][INFO ][o.e.e.NodeEnvironment ] [4zwAMlT] using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [12.8gb], net total_space [112.8gb], types [apfs] [2018-06-28T18:14:19,464][INFO ][o.e.e.NodeEnvironment ] [4zwAMlT] heap size [990.7mb], compressed ordinary object pointers [true] [2018-06-28T18:14:19,467][INFO ][o.e.n.Node ] [4zwAMlT] node name derived from node ID [4zwAMlTzRCaioBeOE9PaNw]; set [node.name] to override [2018-06-28T18:14:19,468][INFO ][o.e.n.Node ] [4zwAMlT] version[6.3.0], pid[33449], build[default/zip/424e937/2018-06-11T23:38:03.357887Z], OS[Mac OS X/10.13.5/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_101/25.101-b13] [2018-06-28T18:14:19,468][INFO ][o.e.n.Node ] [4zwAMlT] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/var/folders/yk/2446sljj6hn82nvzkdgxltmw0000gn/T/elasticsearch.AXb282by, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/local/elasticsearch/6.3.0, -Des.path.conf=/usr/local/elasticsearch/6.3.0/config, -Des.distribution.flavor=default, -Des.distribution.type=zip] [2018-06-28T18:14:19,552][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Unknown properties in plugin descriptor: [jvm, site, isolated] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.3.0.jar:6.3.0] at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.3.0.jar:6.3.0] Caused by: java.lang.IllegalArgumentException: Unknown properties in plugin descriptor: [jvm, site, isolated] at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:240) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.readPluginBundle(PluginsService.java:484) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.findBundles(PluginsService.java:463) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:147) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.0.jar:6.3.0] ... 6 more

这个错误主要看

Caused by: java.lang.IllegalArgumentException: Unknown properties in plugin descriptor: [jvm, site, isolated]

提示，这三个参数错误，那么很简单，直接使用#注释掉即可
另一个错误

[2018-06-28T18:18:49,037][INFO ][o.e.n.Node ] [] initializing ... [2018-06-28T18:18:49,138][INFO ][o.e.e.NodeEnvironment ] [4zwAMlT] using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [12.8gb], net total_space [112.8gb], types [apfs] [2018-06-28T18:18:49,139][INFO ][o.e.e.NodeEnvironment ] [4zwAMlT] heap size [990.7mb], compressed ordinary object pointers [true] [2018-06-28T18:18:49,141][INFO ][o.e.n.Node ] [4zwAMlT] node name derived from node ID [4zwAMlTzRCaioBeOE9PaNw]; set [node.name] to override [2018-06-28T18:18:49,141][INFO ][o.e.n.Node ] [4zwAMlT] version[6.3.0], pid[33593], build[default/zip/424e937/2018-06-11T23:38:03.357887Z], OS[Mac OS X/10.13.5/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_101/25.101-b13] [2018-06-28T18:18:49,142][INFO ][o.e.n.Node ] [4zwAMlT] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/var/folders/yk/2446sljj6hn82nvzkdgxltmw0000gn/T/elasticsearch.k4FNB531, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/local/elasticsearch/6.3.0, -Des.path.conf=/usr/local/elasticsearch/6.3.0/config, -Des.distribution.flavor=default, -Des.distribution.type=zip] [2018-06-28T18:18:53,168][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Plugin [analysis-jieba] was built for Elasticsearch version 6.0.0 but version 6.3.0 is running
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.3.0.jar:6.3.0] at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.3.0.jar:6.3.0] Caused by: java.lang.IllegalArgumentException: Plugin [analysis-jieba] was built for Elasticsearch version 6.0.0 but version 6.3.0 is running
at org.elasticsearch.plugins.PluginsService.verifyCompatibility(PluginsService.java:421) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:618) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:557) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:162) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.0.jar:6.3.0]

还是看那句

Caused by: java.lang.IllegalArgumentException: Plugin [analysis-jieba] was built for Elasticsearch version 6.0.0 but version 6.3.0 is running

版本不对，修复方法也很简单，直接把

elasticsearch.version=6.0.0

改成

elasticsearch.version=6.3.0

修改完成后配置文件如下

# Elasticsearch plugin descriptor file # This file must exist as 'plugin-descriptor.properties' at # the root directory of all plugins. # # A plugin can be 'site', 'jvm', or both. # ### example site plugin for "foo": # # foo.zip <-- zip file for the plugin, with this structure: #   _site/ <-- the contents that will be served #   plugin-descriptor.properties <-- example contents below: # # site=true # description=My cool plugin # version=1.0 # ### example jvm plugin for "foo" # # foo.zip <-- zip file for the plugin, with this structure: #   <arbitrary name1>.jar <-- classes, resources, dependencies #   <arbitrary nameN>.jar <-- any number of jars #   plugin-descriptor.properties <-- example contents below: # # jvm=true # classname=foo.bar.BazPlugin # description=My cool plugin # version=2.0.0-rc1 # elasticsearch.version=2.0 # java.version=1.7 # ### mandatory elements for all plugins: # # 'description': simple summary of the plugin description=A jieba analysis of plugins for Elasticsearch # # 'version': plugin's version version=6.0.0 # # 'name': the plugin name name=analysis-jieba ### mandatory elements for site plugins: # # 'site': set to true to indicate contents of the _site/ #  directory in the root of the plugin should be served. #site=${elasticsearch.plugin.site} # ### mandatory elements for jvm plugins : # # 'jvm': true if the 'classname' class should be loaded #  from jar files in the root directory of the plugin. #  Note that only jar files in the root directory are #  added to the classpath for the plugin! If you need #  other resources, package them into a resources jar. #jvm=true # # 'classname': the name of the class to load, fully-qualified. classname=org.elasticsearch.plugin.analysis.jieba.AnalysisJiebaPlugin # # 'java.version' version of java the code is built against # use the system property java.specification.version # version string must be a sequence of nonnegative decimal integers # separated by "."'s and may have leading zeros java.version=1.8 # # 'elasticsearch.version' version of elasticsearch compiled against # You will have to release a new version of the plugin for each new # elasticsearch release. This version is checked when the plugin # is loaded so Elasticsearch will refuse to start in the presence of # plugins with the incorrect elasticsearch.version. elasticsearch.version=6.3.0 # ### deprecated elements for jvm plugins : # # 'isolated': true if the plugin should have its own classloader. # passing false is deprecated, and only intended to support plugins # that have hard dependencies against each other. If this is # not specified, then the plugin is isolated by default. #isolated=${elasticsearch.plugin.isolated} #

试用结巴中文分词

成功启动后，我们就可以使用 结巴中文 分词插件了
结巴分词插件提供 3 个分析器：jieba_index、jieba_search和jieba_other

所以，我们就可以重新发起前一个章节的请求了

POST http://localhost:9200/_analyze 

请求正文

{ "analyzer" : "jieba_index", "text" : "你好，世界，很高兴认识你！" }

返回响应

{ "tokens" : [ { "token" : "你好", "start_offset" : 0, "end_offset" : 2, "type" : "word", "position" : 0 }, { "token" : "，", "start_offset" : 2, "end_offset" : 3, "type" : "word", "position" : 1 }, { "token" : "世界", "start_offset" : 3, "end_offset" : 5, "type" : "word", "position" : 2 }, { "token" : "，", "start_offset" : 5, "end_offset" : 6, "type" : "word", "position" : 3 }, { "token" : "很", "start_offset" : 6, "end_offset" : 7, "type" : "word", "position" : 4 }, { "token" : "高兴", "start_offset" : 7, "end_offset" : 9, "type" : "word", "position" : 5 }, { "token" : "认识", "start_offset" : 9, "end_offset" : 11, "type" : "word", "position" : 6 }, { "token" : "你", "start_offset" : 11, "end_offset" : 12, "type" : "word", "position" : 7 }, { "token" : "！", "start_offset" : 12, "end_offset" : 13, "type" : "word", "position" : 8 } ] }

返回的分词结果为

你好 ，世界 ，很 高兴 认识 你 ！

至于结巴分词的具体使用我们就不做介绍了，如果你感兴趣，可以访问
1、ElasticSearch自定义分析器-集成结巴分词插件 2、https://github.com/sing1ee/elasticsearch-jieba-plugin