码迷,mamicode.com
首页 > 其他好文 > 详细

压测:celey backend为rabbitmq pk redis

时间:2017-07-01 18:19:26      阅读:201      评论:0      收藏:0      [点我收藏+]

标签:分片   ref   pac   group   压测   print   padding   tip   llb   

使用celery的backend异步获取结果,本文使用rabbitmq 和 redis分别作为backend,代码对比如下

from celery import Celery, platforms
import time
import os
from datetime import datetime

app = Celery(proj,
             broker=amqp://admin:admin@ip:5672,     //rabbitmq
             backend=amqp://admin:admin@ip:5672,    //redis
             //backend=‘redis://ip:10013/0‘,
             include=[tasks]
             )
app.conf.update(
    CELERYD_PREFETCH_MULTIPLIER=1,
    CELERY_ACKS_LAT = True,
)

@app.task
def fun_1(n):
    return {"task_id": "0a14c9ac91de419880c4499a8f57418e",
            "status": 0,
            "desc": str(n) }

@app.task
def chk_total_callback(tasks_rets):
    end_time = datetime.now()
    print "end_time", end_time
    return 0


if __name__ == "__main__":
    app.start()

发送消息

from tasks import *                                                    
from celery import group, chain                                        
from datetime import datetime                                          
import time                                                            
                                                                       
                                                                       
#job_group = group([fun_1.s(i) for i in range(30000)])                 
beg_time = datetime.now()                                              
for j in range(8000):
    job_group = group([fun_1.s(i) for i in range(20)])                 
    chain(job_group, chk_total_callback.s())()                         
    #time.sleep(0.25)                                                  
                                                                       
print "beg_time:", beg_time

压测场景一:

注:1000*20:一次发送1000次,分片20

力度

backend

耗时

内存峰值

均值

1000*20

 

rabbitmq

18:40:36-18:41:21

0.504G

45ms

redis

18:48:37-18:51:21

11.84M

82ms

2000*20

rabbitmq

21:57:06 -21:58:41

1.008G

47.5ms

redis

18:58:52-19:04:10

20.26M

159ms

3000*20

rabbitmq

21:51:40-21:54:03

1.512G

47.6ms

redis

19:43:31-19:51:41

28.18M

163ms

4000*20

rabbitmq

20:59:12-21:02:28

2.268G

49ms

redis

20:00:00-20:10:52

37.10M

163ms

压测场景二:

500*20: 500次,每次间隔1s,分片20

力度

backend

耗时

内存峰值

耗时

500*20

 

rabbitmq

22:34:28-22:43:01

0.504G

513s

redis

22:49:23-22:57:56

7.63M

513s

压测场景三:

8000 * 0.25 = 2000s, 任务数:8000   8000 * 20 + 8000 = 8000 * 21=168000 * 2=336000

力度

backend

耗时

内存峰值

耗时

8000 * 0.25 = 2000s

*20

 

rabbitmq

10:51:38-11:28:06

1.26G

2188s

redis

11:56:17-12:33:00

65.96M

2203s

16000 * 0.25 = 4000s

*20

 

rabbitmq

18:26:39-19:39:37

4.28G

73*60=4380

redis

20:13:57-22:49:33

130.69M

9360s

压测:celey backend为rabbitmq pk redis

标签:分片   ref   pac   group   压测   print   padding   tip   llb   

原文地址:http://www.cnblogs.com/kaituorensheng/p/7102829.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!