concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending 错误 -- Python



在 concurrent.futures 库中有 ThreadPoolExecutor (多线程),ProcessPoolExecutor (多进程)

ThreadPoolExecutor,ProcessPoolExecutor 的区别

ThreadPoolExecutor

ThreadPoolExecutor 多线程并行执行任务,可以共享当前进程变量,但缺点也很致命,由于 python GIL(Global Interpreter Lock 全局解释器锁) 的原因,即使多线程,但其实仍然最多只能占用 CPU 一个核,准确只能说是并发了,如果指定的任务和线程数不恰当(比如一个任务很短,线程数量很多,导致线程频繁调用回收),那么效率还不如单线程

ProcessPoolExecutor

ProcessPoolExecutor 可以使用多核进行计算,但缺点就是进程之间共享数据就比较麻烦,消耗更多的内存。

例子:

注意 ProcessPoolExecutor 使用的位置,否则会造成多进程循环调用,会报

concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending 错误。

from concurrent.futures import ProcessPoolExecutor

URLS = ['http://www.baidu.com', 'http://qq.com', 'http://sina.com']

def task(url, index,timeout=10):
    return index,url

# 在此例子中 if __name__ == "__main__": 一定要加,因为没有 if __name__ 会在创建子进程的时候又会运行,导致错误
if __name__ == "__main__": 
    p = ProcessPoolExecutor(max_workers=3)
    results = p.map(task, URLS,range(3))
    p.shutdown(wait=True)
    for ret,url in results:
        print(ret,url)