as multi-threadings , multi-processes pools in multiprocessing
pool = pool() result = pool.map(func, arg) pool.close() pool.join() why close , join necessary make code safe? bad consequences can make without them?
in loop, it's better put these lines inside or outside loop?
for example,
pool = pool() x in a_ndarray: result = pool.map(func, x) save(result) pool.close() pool.join() and
pool = pool() x in a_ndarray: result = pool.map(func, x) save(result) pool.close() pool.join() i saw others suggested multi-processes cpu-bound tasks , multi-threadings io-bound tasks. disadvantages of applying multi-threading cpu-bound , multi-processes io-bound?
@lee hi folk,
basically, these instructions set closure concepts current executions, "i won't put more data queue(close) , i'll wait end of sub-processes before go on(join)".
from docs:
close()
indicate no more data put on queue current >process. background thread quit once has flushed buffered data pipe. called automatically when queue garbage collected.
join()
block until items in queue have been gotten , processed.
the count of unfinished tasks goes whenever item added queue. count goes down whenever consumer thread calls task_done() indicate item retrieved , work on complete. when count of unfinished tasks drops zero, join() unblocks.
source: python docs
this make better , safer code because use information proper garbage collection , avoid weird or unwanted behavior of code ending main process before child processes end.
for example, if after launch sub-processes call function vary in time execution:
pool = pool() x in a_ndarray: result = pool(func, x) save(result) non_fixed_time_function() #this take 0.1 s or 2 hours. #pool.join() # don't wait child finish if don't wait child finish, in execution want, in other finish 1 child or 2 children, , cause weird results.
about second question, in scenario, take .close() , .join() methods inside loop, before save result.
Comments
Post a Comment