Python多进程并发(multiprocessing)

标签：multiprocessing

 A manager returned by Manager() will support types list, dict, Namespace, Lock, RLock, Semaphore, BoundedSemaphore, Condition, Event, Queue, Value and Array. For example,

from multiprocessing import Process, Manager

def f(d, l):

    d[1] = ‘1‘

    d[‘2‘] = 2

    d[0.25] = None

    l.reverse()

if __name__ == ‘__main__‘:

    manager = Manager()

    d = manager.dict()

    l = manager.list(range(10))

    p = Process(target=f, args=(d, l))

    p.start()

    p.join()

    print d

    print l

will print

{0.25: None, 1: ‘1‘, ‘2‘: 2}

[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

import multiprocessing

import time

def func(msg):

  for i in xrange(3):

    print msg

    time.sleep(1)

if __name__ == "__main__":

  pool = multiprocessing.Pool(processes=4)

  for i in xrange(10):

    msg = "hello %d" %(i)

    pool.apply_async(func, (msg, ))

  pool.close()

  pool.join()

  print "Sub-process(es) done."

使用Pool,关注结果

import multiprocessing

import time

def func(msg):

  for i in xrange(3):

    print msg

    time.sleep(1)

  return "done " + msg

if __name__ == "__main__":

  pool = multiprocessing.Pool(processes=4)

  result = []

  for i in xrange(10):

    msg = "hello %d" %(i)

    result.append(pool.apply_async(func, (msg, )))

  pool.close()

  pool.join()

  for res in result:

    print res.get()

  print "Sub-process(es) done."

#!/usr/bin/env python

#coding=utf-8

"""

Author: Squall

Last modified: 2011-10-18 16:50

Filename: pool.py

Description: a simple sample for pool class

"""

from multiprocessing import Pool

from time import sleep

def f(x):

    for i in range(10):

        print ‘%s --- %s ‘ % (i, x)

        #sleep(1)

def main():

    pool = Pool(processes=3)    # set the processes max number 3

    for i in range(11,20):

        result = pool.apply_async(f, (i,))

    pool.close()

    pool.join()

    if result.successful():

        print ‘successful‘

if __name__ == "__main__":

    main()

 先创建容量为3的进程池，然后将f(i)依次传递给它，运行脚本后利用ps aux | grep pool.py查看进程情况，会发现最多只会有三个进程执行。pool.apply_async()用来向进程池提交目标请求，pool.join()是用来等待进程池中的worker进程执行完毕，防止主进程在worker进程结束前结束。但必pool.join()必须使用在pool.close()或者pool.terminate()之后。其中close()跟terminate()的区别在于close()会等待池中的worker进程执行结束再关闭pool,而terminate()则是直接关闭。result.successful()表示整个调用执行的状态，如果还有worker没有执行完，则会抛出AssertionError异常。

本文出自 “大荒芜经” 博客，请务必保留此出处http://2892931976.blog.51cto.com/5396534/1761762

Python多进程并发(multiprocessing)

标签：multiprocessing

原文地址：http://2892931976.blog.51cto.com/5396534/1761762