Permalink
Cannot retrieve contributors at this time
| PEP: 3145 | |
| Title: Asynchronous I/O For subprocess.Popen | |
| Version: $Revision$ | |
| Last-Modified: $Date$ | |
| Author: (James) Eric Pruitt, Charles R. McCreary, Josiah Carlson | |
| Status: Withdrawn | |
| Type: Standards Track | |
| Content-Type: text/x-rst | |
| Created: 04-Aug-2009 | |
| Python-Version: 3.2 | |
| Post-History: | |
| Abstract | |
| ======== | |
| In its present form, the ``subprocess.Popen`` implementation is prone to | |
| dead-locking and blocking of the parent Python script while waiting on data | |
| from the child process. This PEP proposes to make | |
| ``subprocess.Popen`` more asynchronous to help alleviate these | |
| problems. | |
| PEP Deferral | |
| ============ | |
| Further exploration of the concepts covered in this PEP has been deferred | |
| at least until after PEP 3156 has been resolved. | |
| PEP Withdrawal | |
| ============== | |
| This can be dealt with in the bug tracker. A specific proposal is | |
| attached to [11]_. | |
| Motivation | |
| ========== | |
| A search for "python asynchronous subprocess" will turn up numerous | |
| accounts of people wanting to execute a child process and communicate with | |
| it from time to time reading only the data that is available instead of | |
| blocking to wait for the program to produce data [1]_ [2]_ [3]_. The current | |
| behavior of the ``subprocess`` module is that when a user sends or receives | |
| data via the stdin, stderr and stdout file objects, dead locks are common | |
| and documented [4]_ [5]_. While communicate can be used to alleviate some of | |
| the buffering issues, it will still cause the parent process to block while | |
| attempting to read data when none is available to be read from the child | |
| process. | |
| Rationale | |
| ========= | |
| There is a documented need for asynchronous, non-blocking functionality in | |
| ``subprocess.Popen`` [6]_ [7]_ [2]_ [3]_. Inclusion of the code would improve the | |
| utility of the Python standard library that can be used on Unix based and | |
| Windows builds of Python. Practically every I/O object in Python has a | |
| file-like wrapper of some sort. Sockets already act as such and for | |
| strings there is ``StringIO``. Popen can be made to act like a file by simply | |
| using the methods attached to the ``subprocess.Popen.stderr``, stdout and | |
| stdin file-like objects. But when using the read and write methods of | |
| those options, you do not have the benefit of asynchronous I/O. In the | |
| proposed solution the wrapper wraps the asynchronous methods to mimic a | |
| file object. | |
| Reference Implementation | |
| ======================== | |
| I have been maintaining a Google Code repository that contains all of my | |
| changes including tests and documentation [9]_ as well as blog detailing | |
| the problems I have come across in the development process [10]_. | |
| I have been working on implementing non-blocking asynchronous I/O in the | |
| ``subprocess`` module as well as a wrapper class for ``subprocess.Popen`` | |
| that makes it so that an executed process can take the place of a file by | |
| duplicating all of the methods and attributes that file objects have. | |
| There are two base functions that have been added to the ``subprocess.Popen`` | |
| class: ``Popen.send`` and ``Popen._recv``, each with two separate implementations, | |
| one for Windows and one for Unix-based systems. The Windows | |
| implementation uses ctypes to access the functions needed to control pipes | |
| in the kernel 32 DLL in an asynchronous manner. On Unix based systems, | |
| the Python interface for file control serves the same purpose. The | |
| different implementations of ``Popen.send`` and ``Popen._recv`` have identical | |
| arguments to make code that uses these functions work across multiple | |
| platforms. | |
| When calling the ``Popen._recv`` function, it requires the pipe name be | |
| passed as an argument so there exists the ``Popen.recv`` function that passes | |
| selects stdout as the pipe for ``Popen._recv`` by default. ``Popen.recv_err`` | |
| selects stderr as the pipe by default. ``Popen.recv`` and ``Popen.recv_err`` | |
| are much easier to read and understand than ``Popen._recv('stdout' ...)`` and | |
| ``Popen._recv('stderr' ...)`` respectively. | |
| Since the ``Popen._recv`` function does not wait on data to be produced | |
| before returning a value, it may return empty bytes. ``Popen.asyncread`` | |
| handles this issue by returning all data read over a given time | |
| interval. | |
| The ``ProcessIOWrapper`` class uses the ``asyncread`` and ``asyncwrite`` functions to | |
| allow a process to act like a file so that there are no blocking issues | |
| that can arise from using the stdout and stdin file objects produced from | |
| a ``subprocess.Popen`` call. | |
| References | |
| ========== | |
| .. [1] [ python-Feature Requests-1191964 ] asynchronous Subprocess | |
| https://mail.python.org/pipermail/python-bugs-list/2006-December/ | |
| 036524.html | |
| .. [2] Daily Life in an Ivory Basement : /feb-07/problems-with-subprocess | |
| http://ivory.idyll.org/blog/feb-07/problems-with-subprocess | |
| .. [3] How can I run an external command asynchronously from Python? - Stack | |
| Overflow | |
| http://stackoverflow.com/questions/636561/how-can-i-run-an-external- | |
| command-asynchronously-from-python | |
| .. [4] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation | |
| http://docs.python.org/library/subprocess.html#subprocess.Popen.wait | |
| .. [5] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation | |
| http://docs.python.org/library/subprocess.html#subprocess.Popen.kill | |
| .. [6] Issue 1191964: asynchronous Subprocess - Python tracker | |
| http://bugs.python.org/issue1191964 | |
| .. [7] Module to allow Asynchronous subprocess use on Windows and Posix | |
| platforms - ActiveState Code | |
| http://code.activestate.com/recipes/440554/ | |
| .. [8] subprocess.rst - subprocdev - Project Hosting on Google Code | |
| http://code.google.com/p/subprocdev/source/browse/doc/subprocess.rst?spec=svn2c925e935cad0166d5da85e37c742d8e7f609de5&r=2c925e935cad0166d5da85e37c742d8e7f609de5#437 | |
| .. [9] subprocdev - Project Hosting on Google Code | |
| http://code.google.com/p/subprocdev | |
| .. [10] Python Subprocess Dev | |
| http://subdev.blogspot.com/ | |
| .. [11] https://bugs.python.org/issue18823 -- Idle: use pipes instead of | |
| sockets to talk with user subprocess | |
| Copyright | |
| ========= | |
| This P.E.P. is licensed under the Open Publication License; | |
| http://www.opencontent.org/openpub/. | |