1
多线程 VS 多进程(三)
发布于: 2021 年 05 月 21 日
线程替代方案
subprocess
完全跳过线程,使用进程
是派生进程的主要替代方案
python2.4 后引入
multiprocessiong
使用 threading 接口派生,使用子进程
允许为多核或者多 cpu 派生进程,接口跟 threading 非常相似
python2.6 后引入
concurrent.futures
新的异步执行模块
任务级别的操作
python3.2 后引入
多进程
进程间通讯(InterprocessCommunication, IPC)
进程之间无任何共享状态
进程的创建
直接生成 Process 实例对象,案例 19
import multiprocessing
from time import sleep, ctime
def clock(interval):
while True:
print("The time is %s" % ctime())
sleep(interval)
if __name__ == '__main__':
p = multiprocessing.Process(target=clock, args=(5, ))
p.start()
while True:
print("sleeping......")
sleep(1)
复制代码
sleeping......
The time is Tue Aug 13 19:47:41 2019
sleeping......
sleeping......
sleeping......
sleeping......
sleeping......
The time is Tue Aug 13 19:47:46 2019
sleeping......
sleeping......
sleeping......
sleeping......
...
- 派生子类,案例20
复制代码
import multiprocessing
from time import sleep, ctime
class ClockProcess(multiprocessing.Process):
'''
两个函数比较重要
1. init构造函数
2. run
'''
def __init__(self, interval):
super(ClockProcess, self).__init__()
self.interval = interval
def run(self):
while True:
print("The time is %s" % ctime())
sleep(self.interval)
if __name__ == '__main__':
p = ClockProcess(3)
p.start()
复制代码
The time is Tue Aug 13 19:48:49 2019
The time is Tue Aug 13 19:48:52 2019
The time is Tue Aug 13 19:48:55 2019
The time is Tue Aug 13 19:48:58 2019
The time is Tue Aug 13 19:49:01 2019
The time is Tue Aug 13 19:49:04 2019
...
复制代码
在 os 中查看 pid,ppid 以及他们的关系
案例 21
from multiprocessing import Process
import os
def info(title):
print(title)
print("module name:", __name__)
# 得到父亲进程的id
print("parent process:", os.getppid())
# 得到本身进程的id
print("process id:", os.getpid())
def f(name):
info('function f')
print('hello', name)
if __name__ == '__main__':
info('main line')
p = Process(target=f, args=('bob', ))
p.start()
p.join()
复制代码
main line
module name: __main__
parent process: 11900
process id: 18832
function f
module name: __mp_main__
parent process: 18832
process id: 20868
hello bob
复制代码
生产者消费者模型
JoinableQueue
案例 22
import multiprocessing
from time import ctime
def consumer(input_q):
print("Into consumer:", ctime)
while True:
# 处理项
item = input_q.get()
print('pull', item, 'out of q')
input_q.task_done() # 发出信号通知任务完成
print("Out of consumer:", ctime()) ## 此句未执行,因为q.join()收集到四个task_done()信号后,主进程启动,未等到print此句完成,程序就结束了
def producer(sequence, output_q):
print("Into procuder:", ctime())
for item in sequence:
output_q.put(item)
print('put', item, 'into q')
print('Out of procuder', ctime())
# 建立进程
if __name__ == '__main__':
q = multiprocessing.JoinableQueue()
# 进行消费者进程
cons_p = multiprocessing.Process(target=consumer, args=(q, ))
cons_p.daemon = True
cons_p.start()
# 生产多个项,sequence代表要发送给消费者的项序列
# 在实践中,这可能是生成器的输出或通过一些其他地方生产出来
sequence = [1, 2, 3, 4]
producer(sequence, q)
# 等待所有项被处理
q.join()
复制代码
Into procuder: Tue Aug 13 19:50:38 2019
put 1 into q
put 2 into q
put 3 into q
put 4 into q
Out of procuder Tue Aug 13 19:50:38 2019
Into consumer: <built-in function ctime>
pull 1 out of q
pull 2 out of q
pull 3 out of q
pull 4 out of q
- 队列中哨兵的使用,案例23
复制代码
import multiprocessing
from time import ctime
# 设置哨兵问题
def consumer(input_q):
print("Into consumer:", ctime())
while True:
item = input_q.get()
if item is None:
break
print("pull", item, "out of q")
print ("Out of consumer:", ctime()) ## 此句执行完成,再转入主进程
def producer(sequence, output_q):
print ("Into procuder:", ctime())
for item in sequence:
output_q.put(item)
print ("put", item, "into q")
print ("Out of procuder:", ctime())
if __name__ == '__main__':
q = multiprocessing.Queue()
cons_p = multiprocessing.Process(target = consumer, args = (q,))
cons_p.start()
sequence = [1,2,3,4]
producer(sequence, q)
q.put(None)
cons_p.join()
复制代码
Into procuder: Tue Aug 13 19:51:23 2019
put 1 into q
put 2 into q
put 3 into q
put 4 into q
Out of procuder: Tue Aug 13 19:51:23 2019
Into consumer: Tue Aug 13 19:51:24 2019
pull 1 out of q
pull 2 out of q
pull 3 out of q
pull 4 out of q
Out of consumer: Tue Aug 13 19:51:24 2019
- 哨兵的改进,案例24
复制代码
import multiprocessing
from time import ctime
def consumer(input_q):
print ("Into consumer:", ctime())
while True:
item = input_q.get()
if item is None:
break
print("pull", item, "out of q")
print ("Out of consumer:", ctime())
def producer(sequence, output_q):
for item in sequence:
print ("Into procuder:", ctime())
output_q.put(item)
print ("Out of procuder:", ctime())
if __name__ == '__main__':
q = multiprocessing.Queue()
cons_p1 = multiprocessing.Process (target = consumer, args = (q,))
cons_p1.start()
cons_p2 = multiprocessing.Process (target = consumer, args = (q,))
cons_p2.start()
sequence = [1,2,3,4]
producer(sequence, q)
q.put(None)
q.put(None)
cons_p1.join()
cons_p2.join()
复制代码
Into procuder: Tue Aug 13 19:52:08 2019
Out of procuder: Tue Aug 13 19:52:08 2019
Into procuder: Tue Aug 13 19:52:08 2019
Out of procuder: Tue Aug 13 19:52:08 2019
Into procuder: Tue Aug 13 19:52:08 2019
Out of procuder: Tue Aug 13 19:52:08 2019
Into procuder: Tue Aug 13 19:52:08 2019
Out of procuder: Tue Aug 13 19:52:08 2019
Into consumer: Tue Aug 13 19:52:08 2019
pull 1 out of q
pull 2 out of q
pull 3 out of q
pull 4 out of q
Out of consumer: Tue Aug 13 19:52:08 2019
Into consumer: Tue Aug 13 19:52:08 2019
Out of consumer: Tue Aug 13 19:52:08 2019
复制代码
划线
评论
复制
发布于: 2021 年 05 月 21 日阅读数: 11
版权声明: 本文为 InfoQ 作者【若尘】的原创文章。
原文链接:【http://xie.infoq.cn/article/939e5813b1aca95b5b942dffb】。文章转载请联系作者。
若尘
关注
还未添加个人签名 2021.01.11 加入
还未添加个人简介
评论