标签:form 场景 lin public util vat art ons nts
1.场景:有大数据的数据需要放到首页统计,一般就是聚合、分组之类的,按照年月日进行查询和统计。如果数据量比较少,几十万数据 没什么问题。但是随着数据量的增多,查询速度越来越慢。这个时候就需要去优化了~
刚开始自己的想法是这样的:使用多线程的方式,因为查询每天的数据量很少,那么是不是可以使用多线程的方式,每个线程查询一天的,查询一个月30天,就用30个线程,这样速度会不会快些?
于是,用多线程的方式实现了下。代码如下:
private ExecutorService executorService = new ThreadPoolExecutor(30,30,1, TimeUnit.MILLISECONDS,new LinkedBlockingDeque<>());
public List<Map> getCiServiceBadEvaNumStatistic(SAASIndexQuery saasIndexQuery) throws InvocationTargetException, IllegalAccessException { String startDate = saasIndexQuery.getStartDate(); String endDate = saasIndexQuery.getEndDate(); int days = DateUtil.getDatebetweenOfDayNum(DateUtil.parseDate(startDate,DateUtil.dateFormatPattern),DateUtil.parseDate(endDate,DateUtil.dateFormatPattern)); CompletionService<List<CiOrderStatisticSection>> completionService = new ExecutorCompletionService<List<CiOrderStatisticSection>>(executorService); List<CiOrderStatisticSection> allList = new ArrayList<>(); long start = System.currentTimeMillis(); logger.info("测试异步时间start:" + System.currentTimeMillis()); //CountDownLatch countDownLatch = new CountDownLatch(days); SAASIndexQuery everyDaySaas = new SAASIndexQuery(); BeanUtils.copyProperties(everyDaySaas,saasIndexQuery); for(int i = 0;i<days;i++){ everyDaySaas.setStartDate(DateUtil.afterNDay(saasIndexQuery.getStartDate(),i,DateUtil.dateFormatPattern)); everyDaySaas.setEndDate(DateUtil.afterNDay(everyDaySaas.getStartDate(),1,DateUtil.dateFormatPattern)); //countDownLatch.countDown(); int finalI = i; completionService.submit(new Callable<List<CiOrderStatisticSection>>() { @Override public List<CiOrderStatisticSection> call() throws Exception { //allList.addAll(biSaasCiDeviceDayExMapper.getCiServiceNegativeRate(saasIndexQuery)); //countDownLatch.countDown(); System.out.println("====="+ finalI +"====="); return biSaasCiDeviceDayExMapper.getCiServiceNegativeRate(saasIndexQuery); } }); } System.out.println("==============" + (System.currentTimeMillis()-start) + "毫秒"); long t = System.currentTimeMillis(); for (int i = 0;i<days;i++){ System.out.println("for循环耗时==============+"+i + (System.currentTimeMillis()-t) + "毫秒"); try { Future<List<CiOrderStatisticSection>> future = completionService.take(); List<CiOrderStatisticSection> ciList = future.get(); allList.addAll(ciList); } catch (InterruptedException e) { e.printStackTrace(); } catch (ExecutionException e) { e.printStackTrace(); } } long end = System.currentTimeMillis(); logger.info("测试异步时间end:" + (end-start) + "毫秒"); System.out.println("测试异步时间end:" + (end-start) + "毫秒"); }
测试后发现不对,使用多线程的take方式 每次都会有阻塞,这个阻塞一直没明白是哪里阻塞了? 是线程池、LinkedBlockingDeque 还是for循环 take时候 阻塞了 一直没明白,观察的结果就是每次for循环都要差不多200多毫秒,30个循环要6s多。。。。额,算了 ,还没有原来快呢
2.昨天换了种思路:直接从数据库查询时候做好控制。每次查询先根据月份和年份,查询出来id的最大值和最小值,之后sql里面查询时候加上id在这个最大值和最小是区间内。大概思路是这样:嗯,结果竟然可以。
标签:form 场景 lin public util vat art ons nts
原文地址:https://www.cnblogs.com/thinkingandworkinghard/p/12716695.html