add files
This commit is contained in:
Caleb Fontenot
@@ -0,0 +1,356 @@
|
||||
.. currentmodule:: asyncio
|
||||
|
||||
.. _asyncio-subprocess:
|
||||
|
||||
============
|
||||
Subprocesses
|
||||
============
|
||||
|
||||
This section describes high-level async/await asyncio APIs to
|
||||
create and manage subprocesses.
|
||||
|
||||
.. _asyncio_example_subprocess_shell:
|
||||
|
||||
Here's an example of how asyncio can run a shell command and
|
||||
obtain its result::
|
||||
|
||||
import asyncio
|
||||
|
||||
async def run(cmd):
|
||||
proc = await asyncio.create_subprocess_shell(
|
||||
cmd,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE)
|
||||
|
||||
stdout, stderr = await proc.communicate()
|
||||
|
||||
print(f'[{cmd!r} exited with {proc.returncode}]')
|
||||
if stdout:
|
||||
print(f'[stdout]\n{stdout.decode()}')
|
||||
if stderr:
|
||||
print(f'[stderr]\n{stderr.decode()}')
|
||||
|
||||
asyncio.run(run('ls /zzz'))
|
||||
|
||||
will print::
|
||||
|
||||
['ls /zzz' exited with 1]
|
||||
[stderr]
|
||||
ls: /zzz: No such file or directory
|
||||
|
||||
Because all asyncio subprocess functions are asynchronous and asyncio
|
||||
provides many tools to work with such functions, it is easy to execute
|
||||
and monitor multiple subprocesses in parallel. It is indeed trivial
|
||||
to modify the above example to run several commands simultaneously::
|
||||
|
||||
async def main():
|
||||
await asyncio.gather(
|
||||
run('ls /zzz'),
|
||||
run('sleep 1; echo "hello"'))
|
||||
|
||||
asyncio.run(main())
|
||||
|
||||
See also the `Examples`_ subsection.
|
||||
|
||||
|
||||
Creating Subprocesses
|
||||
=====================
|
||||
|
||||
.. coroutinefunction:: create_subprocess_exec(program, \*args, stdin=None, \
|
||||
stdout=None, stderr=None, loop=None, \
|
||||
limit=None, \*\*kwds)
|
||||
|
||||
Create a subprocess.
|
||||
|
||||
The *limit* argument sets the buffer limit for :class:`StreamReader`
|
||||
wrappers for :attr:`Process.stdout` and :attr:`Process.stderr`
|
||||
(if :attr:`subprocess.PIPE` is passed to *stdout* and *stderr* arguments).
|
||||
|
||||
Return a :class:`~asyncio.subprocess.Process` instance.
|
||||
|
||||
See the documentation of :meth:`loop.subprocess_exec` for other
|
||||
parameters.
|
||||
|
||||
.. coroutinefunction:: create_subprocess_shell(cmd, stdin=None, \
|
||||
stdout=None, stderr=None, loop=None, \
|
||||
limit=None, \*\*kwds)
|
||||
|
||||
Run the *cmd* shell command.
|
||||
|
||||
The *limit* argument sets the buffer limit for :class:`StreamReader`
|
||||
wrappers for :attr:`Process.stdout` and :attr:`Process.stderr`
|
||||
(if :attr:`subprocess.PIPE` is passed to *stdout* and *stderr* arguments).
|
||||
|
||||
Return a :class:`~asyncio.subprocess.Process` instance.
|
||||
|
||||
See the documentation of :meth:`loop.subprocess_shell` for other
|
||||
parameters.
|
||||
|
||||
.. important::
|
||||
|
||||
It is the application's responsibility to ensure that all whitespace and
|
||||
special characters are quoted appropriately to avoid `shell injection
|
||||
<https://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_
|
||||
vulnerabilities. The :func:`shlex.quote` function can be used to properly
|
||||
escape whitespace and special shell characters in strings that are going
|
||||
to be used to construct shell commands.
|
||||
|
||||
.. note::
|
||||
|
||||
The default asyncio event loop implementation on **Windows** does not
|
||||
support subprocesses. Subprocesses are available for Windows if a
|
||||
:class:`ProactorEventLoop` is used.
|
||||
See :ref:`Subprocess Support on Windows <asyncio-windows-subprocess>`
|
||||
for details.
|
||||
|
||||
.. seealso::
|
||||
|
||||
asyncio also has the following *low-level* APIs to work with subprocesses:
|
||||
:meth:`loop.subprocess_exec`, :meth:`loop.subprocess_shell`,
|
||||
:meth:`loop.connect_read_pipe`, :meth:`loop.connect_write_pipe`,
|
||||
as well as the :ref:`Subprocess Transports <asyncio-subprocess-transports>`
|
||||
and :ref:`Subprocess Protocols <asyncio-subprocess-protocols>`.
|
||||
|
||||
|
||||
Constants
|
||||
=========
|
||||
|
||||
.. data:: asyncio.subprocess.PIPE
|
||||
|
||||
Can be passed to the *stdin*, *stdout* or *stderr* parameters.
|
||||
|
||||
If *PIPE* is passed to *stdin* argument, the
|
||||
:attr:`Process.stdin <asyncio.subprocess.Process.stdin>` attribute
|
||||
will point to a :class:`StreamWriter` instance.
|
||||
|
||||
If *PIPE* is passed to *stdout* or *stderr* arguments, the
|
||||
:attr:`Process.stdout <asyncio.subprocess.Process.stdout>` and
|
||||
:attr:`Process.stderr <asyncio.subprocess.Process.stderr>`
|
||||
attributes will point to :class:`StreamReader` instances.
|
||||
|
||||
.. data:: asyncio.subprocess.STDOUT
|
||||
|
||||
Special value that can be used as the *stderr* argument and indicates
|
||||
that standard error should be redirected into standard output.
|
||||
|
||||
.. data:: asyncio.subprocess.DEVNULL
|
||||
|
||||
Special value that can be used as the *stdin*, *stdout* or *stderr* argument
|
||||
to process creation functions. It indicates that the special file
|
||||
:data:`os.devnull` will be used for the corresponding subprocess stream.
|
||||
|
||||
|
||||
Interacting with Subprocesses
|
||||
=============================
|
||||
|
||||
Both :func:`create_subprocess_exec` and :func:`create_subprocess_shell`
|
||||
functions return instances of the *Process* class. *Process* is a high-level
|
||||
wrapper that allows communicating with subprocesses and watching for
|
||||
their completion.
|
||||
|
||||
.. class:: asyncio.subprocess.Process
|
||||
|
||||
An object that wraps OS processes created by the
|
||||
:func:`create_subprocess_exec` and :func:`create_subprocess_shell`
|
||||
functions.
|
||||
|
||||
This class is designed to have a similar API to the
|
||||
:class:`subprocess.Popen` class, but there are some
|
||||
notable differences:
|
||||
|
||||
* unlike Popen, Process instances do not have an equivalent to
|
||||
the :meth:`~subprocess.Popen.poll` method;
|
||||
|
||||
* the :meth:`~asyncio.subprocess.Process.communicate` and
|
||||
:meth:`~asyncio.subprocess.Process.wait` methods don't have a
|
||||
*timeout* parameter: use the :func:`wait_for` function;
|
||||
|
||||
* the :meth:`Process.wait() <asyncio.subprocess.Process.wait>` method
|
||||
is asynchronous, whereas :meth:`subprocess.Popen.wait` method
|
||||
is implemented as a blocking busy loop;
|
||||
|
||||
* the *universal_newlines* parameter is not supported.
|
||||
|
||||
This class is :ref:`not thread safe <asyncio-multithreading>`.
|
||||
|
||||
See also the :ref:`Subprocess and Threads <asyncio-subprocess-threads>`
|
||||
section.
|
||||
|
||||
.. coroutinemethod:: wait()
|
||||
|
||||
Wait for the child process to terminate.
|
||||
|
||||
Set and return the :attr:`returncode` attribute.
|
||||
|
||||
.. note::
|
||||
|
||||
This method can deadlock when using ``stdout=PIPE`` or
|
||||
``stderr=PIPE`` and the child process generates so much output
|
||||
that it blocks waiting for the OS pipe buffer to accept
|
||||
more data. Use the :meth:`communicate` method when using pipes
|
||||
to avoid this condition.
|
||||
|
||||
.. coroutinemethod:: communicate(input=None)
|
||||
|
||||
Interact with process:
|
||||
|
||||
1. send data to *stdin* (if *input* is not ``None``);
|
||||
2. read data from *stdout* and *stderr*, until EOF is reached;
|
||||
3. wait for process to terminate.
|
||||
|
||||
The optional *input* argument is the data (:class:`bytes` object)
|
||||
that will be sent to the child process.
|
||||
|
||||
Return a tuple ``(stdout_data, stderr_data)``.
|
||||
|
||||
If either :exc:`BrokenPipeError` or :exc:`ConnectionResetError`
|
||||
exception is raised when writing *input* into *stdin*, the
|
||||
exception is ignored. This condition occurs when the process
|
||||
exits before all data are written into *stdin*.
|
||||
|
||||
If it is desired to send data to the process' *stdin*,
|
||||
the process needs to be created with ``stdin=PIPE``. Similarly,
|
||||
to get anything other than ``None`` in the result tuple, the
|
||||
process has to be created with ``stdout=PIPE`` and/or
|
||||
``stderr=PIPE`` arguments.
|
||||
|
||||
Note, that the data read is buffered in memory, so do not use
|
||||
this method if the data size is large or unlimited.
|
||||
|
||||
.. method:: send_signal(signal)
|
||||
|
||||
Sends the signal *signal* to the child process.
|
||||
|
||||
.. note::
|
||||
|
||||
On Windows, :py:data:`SIGTERM` is an alias for :meth:`terminate`.
|
||||
``CTRL_C_EVENT`` and ``CTRL_BREAK_EVENT`` can be sent to processes
|
||||
started with a *creationflags* parameter which includes
|
||||
``CREATE_NEW_PROCESS_GROUP``.
|
||||
|
||||
.. method:: terminate()
|
||||
|
||||
Stop the child process.
|
||||
|
||||
On POSIX systems this method sends :py:data:`signal.SIGTERM` to the
|
||||
child process.
|
||||
|
||||
On Windows the Win32 API function :c:func:`TerminateProcess` is
|
||||
called to stop the child process.
|
||||
|
||||
.. method:: kill()
|
||||
|
||||
Kill the child.
|
||||
|
||||
On POSIX systems this method sends :py:data:`SIGKILL` to the child
|
||||
process.
|
||||
|
||||
On Windows this method is an alias for :meth:`terminate`.
|
||||
|
||||
.. attribute:: stdin
|
||||
|
||||
Standard input stream (:class:`StreamWriter`) or ``None``
|
||||
if the process was created with ``stdin=None``.
|
||||
|
||||
.. attribute:: stdout
|
||||
|
||||
Standard output stream (:class:`StreamReader`) or ``None``
|
||||
if the process was created with ``stdout=None``.
|
||||
|
||||
.. attribute:: stderr
|
||||
|
||||
Standard error stream (:class:`StreamReader`) or ``None``
|
||||
if the process was created with ``stderr=None``.
|
||||
|
||||
.. warning::
|
||||
|
||||
Use the :meth:`communicate` method rather than
|
||||
:attr:`process.stdin.write() <stdin>`,
|
||||
:attr:`await process.stdout.read() <stdout>` or
|
||||
:attr:`await process.stderr.read <stderr>`.
|
||||
This avoids deadlocks due to streams pausing reading or writing
|
||||
and blocking the child process.
|
||||
|
||||
.. attribute:: pid
|
||||
|
||||
Process identification number (PID).
|
||||
|
||||
Note that for processes created by the :func:`create_subprocess_shell`
|
||||
function, this attribute is the PID of the spawned shell.
|
||||
|
||||
.. attribute:: returncode
|
||||
|
||||
Return code of the process when it exits.
|
||||
|
||||
A ``None`` value indicates that the process has not terminated yet.
|
||||
|
||||
A negative value ``-N`` indicates that the child was terminated
|
||||
by signal ``N`` (POSIX only).
|
||||
|
||||
|
||||
.. _asyncio-subprocess-threads:
|
||||
|
||||
Subprocess and Threads
|
||||
----------------------
|
||||
|
||||
Standard asyncio event loop supports running subprocesses from
|
||||
different threads, but there are limitations:
|
||||
|
||||
* An event loop must run in the main thread.
|
||||
|
||||
* The child watcher must be instantiated in the main thread
|
||||
before executing subprocesses from other threads. Call the
|
||||
:func:`get_child_watcher` function in the main thread to instantiate
|
||||
the child watcher.
|
||||
|
||||
Note that alternative event loop implementations might not share
|
||||
the above limitations; please refer to their documentation.
|
||||
|
||||
.. seealso::
|
||||
|
||||
The :ref:`Concurrency and multithreading in asyncio
|
||||
<asyncio-multithreading>` section.
|
||||
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
||||
An example using the :class:`~asyncio.subprocess.Process` class to
|
||||
control a subprocess and the :class:`StreamReader` class to read from
|
||||
its standard output.
|
||||
|
||||
.. _asyncio_example_create_subprocess_exec:
|
||||
|
||||
The subprocess is created by the :func:`create_subprocess_exec`
|
||||
function::
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
|
||||
async def get_date():
|
||||
code = 'import datetime; print(datetime.datetime.now())'
|
||||
|
||||
# Create the subprocess; redirect the standard output
|
||||
# into a pipe.
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
sys.executable, '-c', code,
|
||||
stdout=asyncio.subprocess.PIPE)
|
||||
|
||||
# Read one line of output.
|
||||
data = await proc.stdout.readline()
|
||||
line = data.decode('ascii').rstrip()
|
||||
|
||||
# Wait for the subprocess exit.
|
||||
await proc.wait()
|
||||
return line
|
||||
|
||||
if sys.platform == "win32":
|
||||
asyncio.set_event_loop_policy(
|
||||
asyncio.WindowsProactorEventLoopPolicy())
|
||||
|
||||
date = asyncio.run(get_date())
|
||||
print(f"Current date: {date}")
|
||||
|
||||
|
||||
See also the :ref:`same example <asyncio_example_subprocess_proto>`
|
||||
written using low-level APIs.
|
||||
Reference in New Issue
Block a user