标签：elasticsearch 赛克蓝德 日志分析 secilog
普通查询
Elasticsearch java API同时提供了强大的搜索功能，不过这也是很正常的因为所有的http接口到后面都要转换成java代码才可以执行。索引名和type名称都可以是多个，用逗号分开。
SearchRequestBuilder sbuilder = client.prepareSearch("secilog") //index name
.setTypes( "type") //type name
.setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.setQuery(QueryBuilders.termQuery("message", "insert")) // Query
.setPostFilter(QueryBuilders.rangeQuery("eventCount").from(1).to(18))// Filter
.setFrom(0).setSize(60).setExplain(true);
System.out.println(sbuilder.toString());
SearchResponse response = sbuilder.execute().actionGet();
System.out.println(response.toString());
SearchRequestBuilder就是请求的json字符串的解析后的对象，如果想看json数据非常简单，主要把这个对象打印出来就可以，例如上面例子中的System.out.println(sbuilder.toString());
比如我这里打印出的结果见下面，是不是很强大：
{
"from" : 0,
"size" : 60,
"query" : {
"term" : {
"message" : "insert"
}
},
"post_filter" : {
"range" : {
"eventCount" : {
"from" : 1,
"to" : 18,
"include_lower" : true,
"include_upper" : true
}
}
},
"explain" : true
}
SearchResponse是搜索出来的结果，同样可以把这个对象打印出来看一下搜索的结果。下面是我搜索的部分内容：
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 0.15342641,
"hits" : [ {
"_shard" : 0,
"_node" : "jevOQqVQT_a_pAGqKA0p7w",
"_index" : "secilog",
"_type" : "log",
"_id" : "1",
"_score" : 0.15342641,
"_source" : {
"type" : "syslog",
"eventCount" : 1,
"eventDate" : "2016-02-18T06:13:10.818Z",
"message" : "secilog insert doc test"
},
"_explanation" : {
.......
}
} ]
}
}
需要注意的是，上面例子中的参数都可以忽略，例如可以查询全部：
SearchResponse response = client.prepareSearch().execute().actionGet();
查询总数
查询总数和普通查询语法一样，唯一的就是把size设置为0。
SearchRequestBuilder sbuilder = client.prepareSearch("secilog") //index name
.setTypes( "type") //type name
.setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.setQuery(QueryBuilders.termQuery("message", "insert")) // Query
.setPostFilter(QueryBuilders.rangeQuery("eventCount").from(1).to(18)) // Filter
.setFrom(0).setSize(0).setExplain(true);
System.out.println(sbuilder.toString());
SearchResponse response = sbuilder.execute().actionGet();
System.out.println(response.toString());
System.out.println(response.getHits().getTotalHits());
返回的内容：
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 2,
"max_score" : 0.0,
"hits" : [ ]
}
}
滚动查询scroll
具体说明详见文章Elasticsearch 2.20 滚动查询请求，通过java也是比较方便的得到此信息。
QueryBuilder qb = QueryBuilders.termQuery("message", "insert");
SearchRequestBuilder sbuilder = client.prepareSearch(indexName)
.setSearchType(SearchType.SCAN)
.setScroll(new TimeValue(60000))
.setQuery(qb)
.setSize(100);
System.out.println(sbuilder);
//100 hits per shard will be returned for each scroll
SearchResponse scrollResp = sbuilder.execute().actionGet();
while (true) {
for (SearchHit hit : scrollResp.getHits().getHits()) {
System.out.println(hit.getSource());
}
scrollResp = client.prepareSearchScroll(scrollResp.getScrollId())
.setScroll(new TimeValue(60000)).execute().actionGet();
System.out.println(scrollResp);
//Break condition: No hits are returned
if (scrollResp.getHits().getHits().length == 0) {
break;
}
}
我们同样可以打印一下请求的记录和返回的内容。
{
"size" : 100,
"query" : {
"term" : {
"message" : "insert"
}
}
}
{
"_scroll_id" : "c2NhbjswOzE7dG90YWxfaGl0czoyOw==",
"took" : 3,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 2,
"max_score" : 0.0,
"hits" : [ {
"_index" : "secilog",
"_type" : "log",
"_id" : "1",
"_score" : 0.0,
"_source" : {
"type" : "file",
"eventCount" : 1,
"eventDate" : "2016-02-18T06:13:10.818Z",
"message" : "secilog insert doc test"
}
}, {
"_index" : "secilog",
"_type" : "log",
"_id" : "3",
"_score" : 0.0,
"_source" : {
"type" : "syslog",
"eventCount" : 2,
"eventDate" : "2016-02-18T06:19:59.015Z",
"message" : "secilog insert doc test"
}
} ]
}
}
本文由赛克 蓝德(secisland)原创，转载请标明作者和出处。
简单的汇聚查询
SearchRequestBuilder sbuilder = client
.prepareSearch(indexName)
.setQuery(QueryBuilders.matchAllQuery())
.addAggregation(AggregationBuilders.terms("type").field("type"))
.addAggregation(
AggregationBuilders.dateHistogram("eventDate").field("eventDate")
.interval(DateHistogramInterval.YEAR));
System.out.println(sbuilder);
SearchResponse sr = sbuilder.execute().actionGet();
System.out.println(sr);
// Get your facet results
StringTerms agg1 = sr.getAggregations().get("type");
System.out.println(agg1.getBuckets().get(0).getKeyAsString()+":"
+agg1.getBuckets().get(0).getDocCount());
InternalHistogram<?> agg2 = sr.getAggregations().get("eventDate");
List<? extends Histogram.Bucket> bucketList = ((Histogram) agg2).getBuckets();
System.out.println(bucketList.get(0).getKeyAsString()+":"
+bucketList.get(0).getDocCount());
然后我们看一下请求的参数：
{
"query" : {
"match_all" : { }
},
"aggregations" : {
"type" : {
"terms" : {
"field" : "type"
}
},
"eventDate" : {
"date_histogram" : {
"field" : "eventDate",
"interval" : "1y"
}
}
}
}
返回的结果：
{
"took" : 16,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 2,
"max_score" : 1.0,
"hits" : [ {
"_index" : "secilog",
"_type" : "log",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"type" : "file",
"eventCount" : 1,
"eventDate" : "2016-02-18T06:13:10.818Z",
"message" : "secilog insert doc test"
}
}, {
"_index" : "secilog",
"_type" : "log",
"_id" : "3",
"_score" : 1.0,
"_source" : {
"type" : "syslog",
"eventCount" : 2,
"eventDate" : "2016-02-18T06:19:59.015Z",
"message" : "secilog insert doc test"
}
} ]
},
"aggregations" : {
"eventDate" : {
"buckets" : [ {
"key_as_string" : "2016-01-01T00:00:00.000Z",
"key" : 1451606400000,
"doc_count" : 2
} ]
},
"type" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ {
"key" : "file",
"doc_count" : 1
}, {
"key" : "syslog",
"doc_count" : 1
} ]
}
}
}
从上面可以看出，java接口的功能是非常强大的。
赛克蓝德(secisland)后续会逐步对Elasticsearch的最新版本的各项功能进行分析，近请期待。也欢迎加入secisland公众号进行关注。
Elasticsearch 2.2.0 JAVA开发篇：搜索操作
标签：elasticsearch 赛克蓝德 日志分析 secilog
原文地址：http://zhulinu.blog.51cto.com/539189/1743550
