詳解python之多進(jìn)程和進(jìn)程池(Processing庫(kù))

更新時(shí)間：2017年06月09日 10:24:37 作者：今天芒種

本篇文章主要介紹了詳解python之多進(jìn)程和進(jìn)程池(Processing庫(kù))，非常具有實(shí)用價(jià)值，需要的朋友可以參考下

環(huán)境:win7+python2.7

一直想學(xué)習(xí)多進(jìn)程或多線程,但之前只是單純看一點(diǎn)基礎(chǔ)知識(shí)還有簡(jiǎn)單的介紹,無(wú)法理解怎么去應(yīng)用,直到前段時(shí)間看了github的一個(gè)爬蟲(chóng)項(xiàng)目涉及到多進(jìn)程,多線程相關(guān)內(nèi)容,一邊看一邊百度相關(guān)知識(shí)點(diǎn),現(xiàn)在把一些相關(guān)知識(shí)點(diǎn)和一些應(yīng)用寫(xiě)下來(lái)做個(gè)記錄.

首先說(shuō)下什么是進(jìn)程:進(jìn)程是程序在計(jì)算機(jī)上的一次執(zhí)行活動(dòng),當(dāng)運(yùn)行一個(gè)程序的時(shí)候,就啟動(dòng)了一個(gè)進(jìn)程.而進(jìn)程又分為系統(tǒng)進(jìn)程和用戶進(jìn)程.只要是用于完成操作系統(tǒng)的各種功能的進(jìn)程就是系統(tǒng)進(jìn)程,它們就是處于運(yùn)行狀態(tài)下的操作系統(tǒng)本身;而所有由你啟動(dòng)的進(jìn)程都是用戶進(jìn)程。進(jìn)程是操作系統(tǒng)進(jìn)行資源分配的單位。

直觀點(diǎn)說(shuō),在任務(wù)管理器的用戶名上標(biāo)明system的是系統(tǒng)進(jìn)程,標(biāo)明administrator的是用戶進(jìn)程,另外net是網(wǎng)洛,lcacal service是本地服務(wù),關(guān)于進(jìn)程更加具體的信息可以百科,這里得省點(diǎn)力氣,不然收不回了.

一.多進(jìn)程的簡(jiǎn)單使用

如圖,multiprocessing有多個(gè)函數(shù),很多我也還沒(méi)去了解,這里只講我目前了解的.

進(jìn)程創(chuàng)建:Process(target=主要運(yùn)行的函數(shù),name=自定義進(jìn)程名稱可不寫(xiě),args=(參數(shù)))

方法：

is_alive():判斷進(jìn)程是否存活
join([timeout]):子進(jìn)程結(jié)束再執(zhí)行下一步,timeout為超時(shí)時(shí)間,有時(shí)進(jìn)程遇到阻塞,為了程序能夠運(yùn)行下去而設(shè)置超時(shí)時(shí)間
run():如果在創(chuàng)建Process對(duì)象的時(shí)候不指定target，那么就會(huì)默認(rèn)執(zhí)行Process的run方法
start():啟動(dòng)進(jìn)程,區(qū)分run()
terminate():終止進(jìn)程,關(guān)于終止進(jìn)程沒(méi)有這么簡(jiǎn)單,貌似用psutil包會(huì)更好,有機(jī)會(huì)以后了解更多再寫(xiě)下。

其中，Process以start()啟動(dòng)某個(gè)進(jìn)程。

屬性：

authkey: 在文檔中authkey()函數(shù)找到這么一句話:Set authorization key of process設(shè)置過(guò)程的授權(quán)密鑰 ,目前沒(méi)找到相關(guān)應(yīng)用實(shí)例,這個(gè)密鑰是怎么用的呢?文章不提
daemon:父進(jìn)程終止后自動(dòng)終止，且自己不能產(chǎn)生新進(jìn)程，必須在start()之前設(shè)置
exitcode:進(jìn)程在運(yùn)行時(shí)為None、如果為–N，表示被信號(hào)N結(jié)束
name:進(jìn)程的名字,自定義
pid:每個(gè)進(jìn)程有唯一的PID編號(hào)。

1.Process(),start(),join()

# -*- coding:utf-8 -*-
from multiprocessing import Process
import time

def fun1(t):
 print 'this is fun1',time.ctime()
 time.sleep(t)
 print 'fun1 finish',time.ctime()

def fun2(t):
 print 'this is fun2',time.ctime()
 time.sleep(t)
 print 'fun2 finish',time.ctime()

if __name__ == '__main__':
 a=time.time()
 p1=Process(target=fun1,args=(4,))
 p2 = Process(target=fun2, args=(6,))
 p1.start()
 p2.start()
 p1.join()
 p2.join()
 b=time.time()
 print 'finish',b-a

這里一共開(kāi)了兩個(gè)進(jìn)程,p1和p2,arg=(4,)中的4是fun1函數(shù)的參數(shù),這里要用tulpe類型,如果兩個(gè)參數(shù)或更多就是arg=(參數(shù)1,參數(shù)2...),之后用start()啟動(dòng)進(jìn)程,我們?cè)O(shè)置等待p1和p2進(jìn)程結(jié)束再執(zhí)行下一步.來(lái)看下面的運(yùn)行結(jié)果,fun2和fun1基本在同一時(shí)間開(kāi)始運(yùn)行,當(dāng)運(yùn)行完畢(fun1睡眠4秒,同時(shí)fun2睡眠6秒),才執(zhí)行print 'finish',b-a語(yǔ)句

this is fun2 Mon Jun 05 13:48:04 2017
this is fun1 Mon Jun 05 13:48:04 2017
fun1 finish Mon Jun 05 13:48:08 2017
fun2 finish Mon Jun 05 13:48:10 2017
finish 6.20300006866

Process finished with exit code 0

我們?cè)賮?lái)看下start()與join()處于不同位置會(huì)發(fā)生什么

# -*- coding:utf-8 -*-
from multiprocessing import Process
import time

def fun1(t):
 print 'this is fun1',time.ctime()
 time.sleep(t)
 print 'fun1 finish',time.ctime()

def fun2(t):
 print 'this is fun2',time.ctime()
 time.sleep(t)
 print 'fun2 finish',time.ctime()

if __name__ == '__main__':
 a=time.time()
 p1=Process(target=fun1,args=(4,))
 p2 = Process(target=fun2, args=(6,))
 p1.start()
 p1.join()
 p2.start()
 p2.join()
 b=time.time()
 print 'finish',b-a

結(jié)果:

this is fun1 Mon Jun 05 14:19:28 2017
fun1 finish Mon Jun 05 14:19:32 2017
this is fun2 Mon Jun 05 14:19:32 2017
fun2 finish Mon Jun 05 14:19:38 2017
finish 10.1229999065

Process finished with exit code 0

看,現(xiàn)在是先運(yùn)行fun1函數(shù),運(yùn)行完畢再運(yùn)行fun2接著再是print 'finish',即先運(yùn)行進(jìn)程p1再運(yùn)行進(jìn)程p2,感受到j(luò)oin()的魅力了吧.現(xiàn)在再試試注釋掉join()看看又會(huì)出現(xiàn)什么

# -*- coding:utf-8 -*-
from multiprocessing import Process
import time

def fun1(t):
 print 'this is fun1',time.ctime()
 time.sleep(t)
 print 'fun1 finish',time.ctime()

def fun2(t):
 print 'this is fun2',time.ctime()
 time.sleep(t)
 print 'fun2 finish',time.ctime()

if __name__ == '__main__':
 a=time.time()
 p1=Process(target=fun1,args=(4,))
 p2 = Process(target=fun2, args=(6,))
 p1.start()
 p2.start()
 p1.join()
 #p2.join()
 b=time.time()
 print 'finish',b-a

結(jié)果:

this is fun1 Mon Jun 05 14:23:57 2017
this is fun2 Mon Jun 05 14:23:58 2017
fun1 finish Mon Jun 05 14:24:01 2017
finish 4.05900001526
fun2 finish Mon Jun 05 14:24:04 2017

Process finished with exit code 0

這次是運(yùn)行完fun1(因?yàn)閜1進(jìn)程有用join(),所以主程序等待p1運(yùn)行完接著執(zhí)行下一步),接著繼續(xù)運(yùn)行主進(jìn)程的print 'finish',最后fun2運(yùn)行完畢才結(jié)束

2.name,daemon,is_alive():

# -*- coding:utf-8 -*-
from multiprocessing import Process
import time

def fun1(t):
 print 'this is fun1',time.ctime()
 time.sleep(t)
 print 'fun1 finish',time.ctime()

def fun2(t):
 print 'this is fun2',time.ctime()
 time.sleep(t)
 print 'fun2 finish',time.ctime()

if __name__ == '__main__':
 a=time.time()
 p1=Process(name='fun1進(jìn)程',target=fun1,args=(4,))
 p2 = Process(name='fun2進(jìn)程',target=fun2, args=(6,))
 p1.daemon=True
 p2.daemon = True
 p1.start()
 p2.start()
 p1.join()
 print p1,p2
 print '進(jìn)程1:',p1.is_alive(),'進(jìn)程2:',p2.is_alive()
 #p2.join()
 b=time.time()
 print 'finish',b-a

結(jié)果:

this is fun2 Mon Jun 05 14:43:49 2017
this is fun1 Mon Jun 05 14:43:49 2017
fun1 finish Mon Jun 05 14:43:53 2017
<Process(fun1進(jìn)程, stopped daemon)> <Process(fun2進(jìn)程, started daemon)>
進(jìn)程1: False 進(jìn)程2: True
finish 4.06500005722

Process finished with exit code 0

可以看到,name是給進(jìn)程賦予名字, 運(yùn)行到print '進(jìn)程1:',p1.is_alive(),'進(jìn)程2:',p2.is_alive() 這句的時(shí)候,p1進(jìn)程已經(jīng)結(jié)束(返回False),p2進(jìn)程仍然在運(yùn)行(返回True),但p2沒(méi)有用join(),所以直接接著執(zhí)行主進(jìn)程,由于用了daemon=Ture,父進(jìn)程終止后自動(dòng)終止,p2進(jìn)程沒(méi)有結(jié)束就強(qiáng)行結(jié)束整個(gè)程序了.

3.run()

run()在Process沒(méi)有指定target函數(shù)時(shí),默認(rèn)用run()函數(shù)運(yùn)行程序,

# -*- coding:utf-8 -*-
from multiprocessing import Process
import time

def fun1(t):
 print 'this is fun1',time.ctime()
 time.sleep(t)
 print 'fun1 finish',time.ctime()

def fun2(t):
 print 'this is fun2',time.ctime()
 time.sleep(t)
 print 'fun2 finish',time.ctime()

if __name__ == '__main__':
 a = time.time()
 p=Process()
 p.start()
 p.join()
 b = time.time()
 print 'finish', b - a

結(jié)果:

finish 0.0840001106262

從結(jié)果看出,進(jìn)程p什么也沒(méi)做,為了讓進(jìn)程正常運(yùn)行,我們醬紫寫(xiě):

目標(biāo)函數(shù)沒(méi)有參數(shù):

# -*- coding:utf-8 -*-
from multiprocessing import Process
import time

def fun1():
 print 'this is fun1',time.ctime()
 time.sleep(2)
 print 'fun1 finish',time.ctime()

def fun2(t):
 print 'this is fun2',time.ctime()
 time.sleep(t)
 print 'fun2 finish',time.ctime()

if __name__ == '__main__':
 a = time.time()
 p=Process()
 p.run=fun1
 p.start()
 p.join()
 b = time.time()
 print 'finish', b - a

結(jié)果:

this is fun1 Mon Jun 05 16:34:41 2017
fun1 finish Mon Jun 05 16:34:43 2017
finish 2.11500000954

Process finished with exit code 0

目標(biāo)函數(shù)有參數(shù):

# -*- coding:utf-8 -*-
from multiprocessing import Process
import time

def fun1(t):
 print 'this is fun1',time.ctime()
 time.sleep(t)
 print 'fun1 finish',time.ctime()

def fun2(t):
 print 'this is fun2',time.ctime()
 time.sleep(t)
 print 'fun2 finish',time.ctime()

if __name__ == '__main__':
 a = time.time()
 p=Process()
 p.run=fun1(2)
 p.start()
 p.join()
 b = time.time()
 print 'finish', b - a

結(jié)果:

this is fun1 Mon Jun 05 16:36:27 2017
fun1 finish Mon Jun 05 16:36:29 2017
Process Process-1:
Traceback (most recent call last):
 File "E:\Anaconda2\lib\multiprocessing\process.py", line 258, in _bootstrap
 self.run()
TypeError: 'NoneType' object is not callable
finish 2.0529999733

Process finished with exit code 0

目標(biāo)函數(shù)有參數(shù)的出現(xiàn)了異常,為什么呢?我現(xiàn)在還找不到原因,但是實(shí)踐發(fā)現(xiàn),當(dāng)最后一個(gè)參數(shù)賦予進(jìn)程運(yùn)行后,沒(méi)有其他參數(shù),就會(huì)出現(xiàn)這個(gè)異常,有人知道的望告知.

二.進(jìn)程池

對(duì)于需要使用幾個(gè)甚至十幾個(gè)進(jìn)程時(shí),我們使用Process還是比較方便的,但是如果要成百上千個(gè)進(jìn)程,用Process顯然太笨了,multiprocessing提供了Pool類,即現(xiàn)在要講的進(jìn)程池,能夠?qū)⒈姸噙M(jìn)程放在一起,設(shè)置一個(gè)運(yùn)行進(jìn)程上限,每次只運(yùn)行設(shè)置的進(jìn)程數(shù),等有進(jìn)程結(jié)束,再添加新的進(jìn)程

Pool(processes =num):設(shè)置運(yùn)行進(jìn)程數(shù),當(dāng)一個(gè)進(jìn)程運(yùn)行完,會(huì)添加新的進(jìn)程進(jìn)去

apply_async(函數(shù),(參數(shù))):非阻塞,其中參數(shù)是tulpe類型,

apply(函數(shù),(參數(shù))):阻塞

close():關(guān)閉pool,不能再添加新的任務(wù)

terminate():結(jié)束運(yùn)行的進(jìn)程，不再處理未完成的任務(wù)

join():和Process介紹的作用一樣, 但要在close或terminate之后使用。

1.單個(gè)進(jìn)程池

# -*- coding:utf-8 -*-
from multiprocessing import Pool
import time

def fun1(t):
 print 'this is fun1',time.ctime()
 time.sleep(t)
 print 'fun1 finish',time.ctime()

def fun2(t):
 print 'this is fun2',time.ctime()
 time.sleep(t)
 print 'fun2 finish',time.ctime()

if __name__ == '__main__':
 a=time.time()
 pool = Pool(processes =3) # 可以同時(shí)跑3個(gè)進(jìn)程
 for i in range(3,8):
  pool.apply_async(fun1,(i,))
 pool.close()
 pool.join()
 b=time.time()
 print 'finish',b-a

結(jié)果:

this is fun1 Mon Jun 05 15:15:38 2017
this is fun1 Mon Jun 05 15:15:38 2017
this is fun1 Mon Jun 05 15:15:38 2017
fun1 finish Mon Jun 05 15:15:41 2017
this is fun1 Mon Jun 05 15:15:41 2017
fun1 finish Mon Jun 05 15:15:42 2017
this is fun1 Mon Jun 05 15:15:42 2017
fun1 finish Mon Jun 05 15:15:43 2017
fun1 finish Mon Jun 05 15:15:47 2017
fun1 finish Mon Jun 05 15:15:49 2017
finish 11.1370000839

Process finished with exit code 0

從上面的結(jié)果可以看到,設(shè)置了3個(gè)運(yùn)行進(jìn)程上限,15:15:38這個(gè)時(shí)間同時(shí)開(kāi)始三個(gè)進(jìn)程,當(dāng)?shù)谝粋€(gè)進(jìn)程結(jié)束時(shí)(參數(shù)為3秒那個(gè)進(jìn)程),會(huì)添加新的進(jìn)程,如此循環(huán),直至進(jìn)程池運(yùn)行完再執(zhí)行主進(jìn)程語(yǔ)句b=time.time() print 'finish',b-a .這里用到非阻塞apply_async(),再來(lái)對(duì)比下阻塞apply()

# -*- coding:utf-8 -*-
from multiprocessing import Pool
import time

def fun1(t):
 print 'this is fun1',time.ctime()
 time.sleep(t)
 print 'fun1 finish',time.ctime()

def fun2(t):
 print 'this is fun2',time.ctime()
 time.sleep(t)
 print 'fun2 finish',time.ctime()

if __name__ == '__main__':
 a=time.time()
 pool = Pool(processes =3) # 可以同時(shí)跑3個(gè)進(jìn)程
 for i in range(3,8):
  pool.apply(fun1,(i,))
 pool.close()
 pool.join()
 b=time.time()
 print 'finish',b-a

結(jié)果:

this is fun1 Mon Jun 05 15:59:26 2017
fun1 finish Mon Jun 05 15:59:29 2017
this is fun1 Mon Jun 05 15:59:29 2017
fun1 finish Mon Jun 05 15:59:33 2017
this is fun1 Mon Jun 05 15:59:33 2017
fun1 finish Mon Jun 05 15:59:38 2017
this is fun1 Mon Jun 05 15:59:38 2017
fun1 finish Mon Jun 05 15:59:44 2017
this is fun1 Mon Jun 05 15:59:44 2017
fun1 finish Mon Jun 05 15:59:51 2017
finish 25.1610000134

Process finished with exit code 0

可以看到,阻塞是當(dāng)一個(gè)進(jìn)程結(jié)束后,再進(jìn)行下一個(gè)進(jìn)程,一般我們都用非阻塞apply_async()

2.多個(gè)進(jìn)程池

上面是使用單個(gè)進(jìn)程池的,對(duì)于多個(gè)進(jìn)程池,我們可以用for循環(huán),直接看代碼

# -*- coding:utf-8 -*-
from multiprocessing import Pool
import time

def fun1(t):
 print 'this is fun1',time.ctime()
 time.sleep(t)
 print 'fun1 finish',time.ctime()

def fun2(t):
 print 'this is fun2',time.ctime()
 time.sleep(t)
 print 'fun2 finish',time.ctime()

if __name__ == '__main__':
 a=time.time()
 pool = Pool(processes =3) # 可以同時(shí)跑3個(gè)進(jìn)程
 for fun in [fun1,fun2]:
  for i in range(3,8):
   pool.apply_async(fun,(i,))
 pool.close()
 pool.join()
 b=time.time()
 print 'finish',b-a

結(jié)果:

this is fun1 Mon Jun 05 16:04:38 2017
this is fun1 Mon Jun 05 16:04:38 2017
this is fun1 Mon Jun 05 16:04:38 2017
fun1 finish Mon Jun 05 16:04:41 2017
this is fun1 Mon Jun 05 16:04:41 2017
fun1 finish Mon Jun 05 16:04:42 2017
this is fun1 Mon Jun 05 16:04:42 2017
fun1 finish Mon Jun 05 16:04:43 2017
this is fun2 Mon Jun 05 16:04:43 2017
fun2 finish Mon Jun 05 16:04:46 2017
this is fun2 Mon Jun 05 16:04:46 2017
fun1 finish Mon Jun 05 16:04:47 2017
this is fun2 Mon Jun 05 16:04:47 2017
fun1 finish Mon Jun 05 16:04:49 2017
this is fun2 Mon Jun 05 16:04:49 2017
fun2 finish Mon Jun 05 16:04:50 2017
this is fun2 Mon Jun 05 16:04:50 2017
fun2 finish Mon Jun 05 16:04:52 2017
fun2 finish Mon Jun 05 16:04:55 2017
fun2 finish Mon Jun 05 16:04:57 2017
finish 19.1670000553

Process finished with exit code 0

看到了,在fun1運(yùn)行完接著運(yùn)行fun2.

另外對(duì)于沒(méi)有參數(shù)的情況,就直接 pool.apply_async(funtion),無(wú)需寫(xiě)上參數(shù).

在學(xué)習(xí)編寫(xiě)程序過(guò)程,曾遇到不用if _name_ == '_main_':而直接運(yùn)行程序,這樣結(jié)果會(huì)出錯(cuò),經(jīng)查詢,在Windows上要想使用進(jìn)程模塊，就必須把有關(guān)進(jìn)程的代碼寫(xiě)在當(dāng)前.py文件的if _name_ == ‘_main_' :語(yǔ)句的下面，才能正常使用Windows下的進(jìn)程模塊。Unix/Linux下則不需要。原因有人這么說(shuō):在執(zhí)行的時(shí)候，由于你寫(xiě)的 py 會(huì)被當(dāng)成module 讀進(jìn)執(zhí)行。所以，一定要判斷自身是否為 _main_。也就是要：

if __name__ == ‘__main__' :
# do something.

這里我自己還搞不清楚,期待以后能夠理解

學(xué)習(xí)的過(guò)程中,還涉及了經(jīng)常和進(jìn)程一起運(yùn)用的隊(duì)列Queue和線程threading,有時(shí)間以后再寫(xiě)吧，希望對(duì)大家的學(xué)習(xí)有所幫助，也希望大家多多支持腳本之家。

您可能感興趣的文章:

最新国产好看的视频,伊人天堂AV在线,国产Aaaaaa视频,蜜臀视频在线观看一区,人妻av色图,密臀久久久精品影片,青青视频免费观看毛片,久草在线观看视,国产三级精品色情在线

詳解python之多進(jìn)程和進(jìn)程池(Processing庫(kù))

相關(guān)文章

最新評(píng)論

大家感興趣的內(nèi)容

最近更新的內(nèi)容

常用在線小工具