How can I handle KeyboardInterrupt events with python’s multiprocessing Pools? Here is a simple example:
from multiprocessing import Pool from time import sleep from sys import exit def slowly_square(i): sleep(1) return i*i def go(): pool = Pool(8) try: results = pool.map(slowly_square, range(40)) except KeyboardInterrupt: # **** THIS PART NEVER EXECUTES. **** pool.terminate() print "You cancelled the program!" sys.exit(1) print "\nFinally, here are the results: ", results if __name__ == "__main__": go()
When running the code above, the
KeyboardInterrupt gets raised when I press
^C, but the process simply hangs at that point and I have to kill it externally.
I want to be able to press
^C at any time and cause all of the processes to exit gracefully.
This is a Python bug. When waiting for a condition in threading.Condition.wait(), KeyboardInterrupt is never sent. Repro:
import threading cond = threading.Condition(threading.Lock()) cond.acquire() cond.wait(None) print "done"
The KeyboardInterrupt exception won’t be delivered until wait() returns, and it never returns, so the interrupt never happens. KeyboardInterrupt should almost certainly interrupt a condition wait.
Note that this doesn’t happen if a timeout is specified; cond.wait(1) will receive the interrupt immediately. So, a workaround is to specify a timeout. To do that, replace
results = pool.map(slowly_square, range(40))
results = pool.map_async(slowly_square, range(40)).get(9999999)
Answered By – Glenn Maynard