标签：
常用的Lucene查询代码如下所示，该代码的作用是将path路径下的所有索引信息返回
1 public String matchAll(String path) { 2 try { 3 Directory directory = FSDirectory.open(new File(path)); 4 DirectoryReader reader = DirectoryReader.open(directory); 5 IndexSearcher searcher = new IndexSearcher(reader); 6 MatchAllDocsQuery query = new MatchAllDocsQuery(); 7 8 ScoreDoc[] hits = searcher.search(query, null, Integer.MAX_VALUE).scoreDocs; 9 StringBuffer buffer = new StringBuffer(); 10 for (int i = 0; i < hits.length; i++) { 11 Document hitDocument = searcher.doc(hits[i].doc); 12 // System.out.println(hitDocument.get("key") 13 // + "......"+hitDocument.get("value")); 14 buffer.append(hitDocument.get("key")+";"+hitDocument.get("value")+"|"); 15 } 16 return buffer.toString(); 17 } catch (IOException e) { 18 e.printStackTrace(); 19 } 20 return null; 21 }
但是当该文件夹下索引的数目比较巨大，那么在执行以下代码的时候，则会出现java.lang.OutOfMemoryError: Java heap space的提示
ScoreDoc[] hits = searcher.search(query, null, Integer.MAX_VALUE).scoreDocs;
这时候，我们可以考虑使用分页技术，比如以前大约1亿条数据，我们可以将其分成100个100W的页，每次对100W条索引数据进行处理，这样就可以避免上述情况的发生。在Lucene 中，我们使用searchAfter的方法实现上述功能。它的官方API介绍如下所示：
public TopDocs searchAfter(ScoreDoc after, Query query, int n) throws IOException
n hits for query where all results are after a previous result (after).
By passing the bottom result from a previous page as after, this method can be used for efficient ‘deep-paging‘ across potentially large result sets.
1 private String transToContent(IndexSearcher searcher,TopDocs topDocs) throws IOException { 2 ScoreDoc[] scoreDocs = topDocs.scoreDocs; 3 StringBuffer sb = new StringBuffer(); 4 for(int i=0; i<scoreDocs.length; i++) { 5 Document doc = searcher.doc(scoreDocs[i].doc); 6 sb.append(doc.get("key")+";"+doc.get("value")+"|"); 7 } 8 return sb.toString(); 9 } 10 11 private void matchAll(String path) { 12 try { 13 Directory directory = FSDirectory.open(new File(path)); 14 DirectoryReader reader = DirectoryReader.open(directory); 15 IndexSearcher searcher = new IndexSearcher(reader); 16 17 ScoreDoc after = null; 18 TopDocs topDocs = searcher.searchAfter(after, new MatchAllDocsQuery(), Preference.PAGE_SIZE); 19 int curPage = 1; 20 while(topDocs.scoreDocs.length > 0) { 21 System.out.println("Current Page:"+ (curPage++) ); 22 System.out.println(transToContent(searcher, topDocs)); 23 after = topDocs.scoreDocs[topDocs.scoreDocs.length -1]; 24 topDocs = searcher.searchAfter(after, new MatchAllDocsQuery(), Preference.PAGE_SIZE); 25 } 26 } catch (IOException e) { 27 e.printStackTrace(); 28 } 29 }
标签：
原文地址：http://www.cnblogs.com/nashiyue/p/4652506.html
