python subprocess.Popen hanging

 child = subprocess.Popen(command, shell=True, env=environment, close_fds=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, ) subout = "" with child.stdout: for line in iter(child.stdout.readline, b''): subout += line logging.info(subout) rc = child.wait()

some times (intermittently) this hangs forever. not sure if it hangs on iter(child.stdout.readline) or child.wait()

i ps -ef for the process it Popens and that process no longer exists

my guess is that it has do with bufsize so that child.stdout.readline is going on forever but i have no idea how to test it and as this happens intermittently

I could implement alarm but i m not sure if that's appropriate as i cant really tell whether the popen'd process is just slow or hanging

let's assume that either child.stdout.readline or wait() hangs forever, what actions could i take besides alarm ?

2 Answers

You're likely hitting the deadlock that's explained in the documentation:

Popen.wait():

Wait for child process to terminate. Set and return returncode attribute.

Warning: This will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.

The solution is to use Popen.communicate().

Add close_fds=True to the call to subprocess.Popen() as mentioned here.