码迷,mamicode.com
首页 > 其他好文 > 详细

大数据拆分处理方式

时间:2016-03-03 16:17:07      阅读:153      评论:0      收藏:0      [点我收藏+]

标签:

思路:
1:参考分页方式将数据拆成指定大小线程数
2:在每个独立的线程中去读取数据并处理数据
步骤一实现

public class IndexIntiTools {
    
    public static AtomicInteger runflag=new AtomicInteger();

    //用于测试
    public static List<Object>  syncList=new CopyOnWriteArrayList<Object>();
    private static int idxThreadCount = 10;

    private static Executor ex = Executors.newFixedThreadPool(idxThreadCount);

    /**
     * 构建索引
     * @param hql
     * @param size
     */
    public static void build(String hql, int size) {
        int pagecount = idxThreadCount;
        int count = size / pagecount;
        int mod = size % pagecount;
        List<Runnable> runList = new ArrayList<Runnable>(pagecount);
        IndexExecutor idxExecutor;
        for (int i = 0; i < pagecount; i++) {
            if (i == (pagecount - 1)) {
                idxExecutor = new IndexExecutor(hql, i * count, count + mod);
            } else {
                idxExecutor = new IndexExecutor(hql, i * count, count);
            }
            runList.add(idxExecutor);
        }
        for (Runnable runnable : runList) {
            runflag.incrementAndGet();
            ex.execute(runnable);
        }
    }

}

步骤二实现

public class IndexExecutor implements Runnable{
    
    private static final Log log = LogFactory.getLog(IndexExecutor.class);
    private int start;
    private int limit;
    private String hql;
    public IndexExecutor(String hql,int start, int limit) {
        this.hql=hql;
        this.start = start;
        this.limit = limit;
    }

    @Override
    public void run() {
        log.info("hql:"+hql+",start:"+start+",limit"+limit);
        //查询数据库(hql,start,limit);
        log.info(list);
        IndexIntiTools.syncList.addAll(list);
        IndexIntiTools.runflag.decrementAndGet();
    }


}

大数据拆分处理方式

标签:

原文地址:http://www.cnblogs.com/xiongmaotailang/p/5239012.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!