3397 lines
360 KiB
HTML
3397 lines
360 KiB
HTML
|
||
<!DOCTYPE html>
|
||
|
||
<html xmlns="http://www.w3.org/1999/xhtml">
|
||
<head>
|
||
<meta charset="utf-8" />
|
||
<title>multiprocessing — Process-based parallelism — Python 3.7.4 documentation</title>
|
||
<link rel="stylesheet" href="../_static/pydoctheme.css" type="text/css" />
|
||
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
|
||
|
||
<script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
|
||
<script type="text/javascript" src="../_static/jquery.js"></script>
|
||
<script type="text/javascript" src="../_static/underscore.js"></script>
|
||
<script type="text/javascript" src="../_static/doctools.js"></script>
|
||
<script type="text/javascript" src="../_static/language_data.js"></script>
|
||
|
||
<script type="text/javascript" src="../_static/sidebar.js"></script>
|
||
|
||
<link rel="search" type="application/opensearchdescription+xml"
|
||
title="Search within Python 3.7.4 documentation"
|
||
href="../_static/opensearch.xml"/>
|
||
<link rel="author" title="About these documents" href="../about.html" />
|
||
<link rel="index" title="Index" href="../genindex.html" />
|
||
<link rel="search" title="Search" href="../search.html" />
|
||
<link rel="copyright" title="Copyright" href="../copyright.html" />
|
||
<link rel="next" title="The concurrent package" href="concurrent.html" />
|
||
<link rel="prev" title="threading — Thread-based parallelism" href="threading.html" />
|
||
<link rel="shortcut icon" type="image/png" href="../_static/py.png" />
|
||
<link rel="canonical" href="https://docs.python.org/3/library/multiprocessing.html" />
|
||
|
||
<script type="text/javascript" src="../_static/copybutton.js"></script>
|
||
<script type="text/javascript" src="../_static/switchers.js"></script>
|
||
|
||
|
||
|
||
<style>
|
||
@media only screen {
|
||
table.full-width-table {
|
||
width: 100%;
|
||
}
|
||
}
|
||
</style>
|
||
|
||
|
||
</head><body>
|
||
|
||
<div class="related" role="navigation" aria-label="related navigation">
|
||
<h3>Navigation</h3>
|
||
<ul>
|
||
<li class="right" style="margin-right: 10px">
|
||
<a href="../genindex.html" title="General Index"
|
||
accesskey="I">index</a></li>
|
||
<li class="right" >
|
||
<a href="../py-modindex.html" title="Python Module Index"
|
||
>modules</a> |</li>
|
||
<li class="right" >
|
||
<a href="concurrent.html" title="The concurrent package"
|
||
accesskey="N">next</a> |</li>
|
||
<li class="right" >
|
||
<a href="threading.html" title="threading — Thread-based parallelism"
|
||
accesskey="P">previous</a> |</li>
|
||
<li><img src="../_static/py.png" alt=""
|
||
style="vertical-align: middle; margin-top: -1px"/></li>
|
||
<li><a href="https://www.python.org/">Python</a> »</li>
|
||
<li>
|
||
<span class="language_switcher_placeholder">en</span>
|
||
<span class="version_switcher_placeholder">3.7.4</span>
|
||
<a href="../index.html">Documentation </a> »
|
||
</li>
|
||
|
||
<li class="nav-item nav-item-1"><a href="index.html" >The Python Standard Library</a> »</li>
|
||
<li class="nav-item nav-item-2"><a href="concurrency.html" accesskey="U">Concurrent Execution</a> »</li>
|
||
<li class="right">
|
||
|
||
|
||
<div class="inline-search" style="display: none" role="search">
|
||
<form class="inline-search" action="../search.html" method="get">
|
||
<input placeholder="Quick search" type="text" name="q" />
|
||
<input type="submit" value="Go" />
|
||
<input type="hidden" name="check_keywords" value="yes" />
|
||
<input type="hidden" name="area" value="default" />
|
||
</form>
|
||
</div>
|
||
<script type="text/javascript">$('.inline-search').show(0);</script>
|
||
|
|
||
</li>
|
||
|
||
</ul>
|
||
</div>
|
||
|
||
<div class="document">
|
||
<div class="documentwrapper">
|
||
<div class="bodywrapper">
|
||
<div class="body" role="main">
|
||
|
||
<div class="section" id="module-multiprocessing">
|
||
<span id="multiprocessing-process-based-parallelism"></span><h1><a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a> — Process-based parallelism<a class="headerlink" href="#module-multiprocessing" title="Permalink to this headline">¶</a></h1>
|
||
<p><strong>Source code:</strong> <a class="reference external" href="https://github.com/python/cpython/tree/3.7/Lib/multiprocessing/">Lib/multiprocessing/</a></p>
|
||
<hr class="docutils" />
|
||
<div class="section" id="introduction">
|
||
<h2>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h2>
|
||
<p><a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a> is a package that supports spawning processes using an
|
||
API similar to the <a class="reference internal" href="threading.html#module-threading" title="threading: Thread-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">threading</span></code></a> module. The <a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a> package
|
||
offers both local and remote concurrency, effectively side-stepping the
|
||
<a class="reference internal" href="../glossary.html#term-global-interpreter-lock"><span class="xref std std-term">Global Interpreter Lock</span></a> by using subprocesses instead of threads. Due
|
||
to this, the <a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a> module allows the programmer to fully
|
||
leverage multiple processors on a given machine. It runs on both Unix and
|
||
Windows.</p>
|
||
<p>The <a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a> module also introduces APIs which do not have
|
||
analogs in the <a class="reference internal" href="threading.html#module-threading" title="threading: Thread-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">threading</span></code></a> module. A prime example of this is the
|
||
<a class="reference internal" href="#multiprocessing.pool.Pool" title="multiprocessing.pool.Pool"><code class="xref py py-class docutils literal notranslate"><span class="pre">Pool</span></code></a> object which offers a convenient means of
|
||
parallelizing the execution of a function across multiple input values,
|
||
distributing the input data across processes (data parallelism). The following
|
||
example demonstrates the common practice of defining such functions in a module
|
||
so that child processes can successfully import that module. This basic example
|
||
of data parallelism using <a class="reference internal" href="#multiprocessing.pool.Pool" title="multiprocessing.pool.Pool"><code class="xref py py-class docutils literal notranslate"><span class="pre">Pool</span></code></a>,</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Pool</span>
|
||
|
||
<span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="n">x</span><span class="o">*</span><span class="n">x</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="k">with</span> <span class="n">Pool</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span> <span class="k">as</span> <span class="n">p</span><span class="p">:</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]))</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>will print to standard output</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">9</span><span class="p">]</span>
|
||
</pre></div>
|
||
</div>
|
||
<div class="section" id="the-process-class">
|
||
<h3>The <a class="reference internal" href="#multiprocessing.Process" title="multiprocessing.Process"><code class="xref py py-class docutils literal notranslate"><span class="pre">Process</span></code></a> class<a class="headerlink" href="#the-process-class" title="Permalink to this headline">¶</a></h3>
|
||
<p>In <a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a>, processes are spawned by creating a <a class="reference internal" href="#multiprocessing.Process" title="multiprocessing.Process"><code class="xref py py-class docutils literal notranslate"><span class="pre">Process</span></code></a>
|
||
object and then calling its <a class="reference internal" href="#multiprocessing.Process.start" title="multiprocessing.Process.start"><code class="xref py py-meth docutils literal notranslate"><span class="pre">start()</span></code></a> method. <a class="reference internal" href="#multiprocessing.Process" title="multiprocessing.Process"><code class="xref py py-class docutils literal notranslate"><span class="pre">Process</span></code></a>
|
||
follows the API of <a class="reference internal" href="threading.html#threading.Thread" title="threading.Thread"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.Thread</span></code></a>. A trivial example of a
|
||
multiprocess program is</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Process</span>
|
||
|
||
<span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="n">p</span> <span class="o">=</span> <span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">f</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="s1">'bob'</span><span class="p">,))</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>To show the individual process IDs involved, here is an expanded example:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Process</span>
|
||
<span class="kn">import</span> <span class="nn">os</span>
|
||
|
||
<span class="k">def</span> <span class="nf">info</span><span class="p">(</span><span class="n">title</span><span class="p">):</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">title</span><span class="p">)</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'module name:'</span><span class="p">,</span> <span class="vm">__name__</span><span class="p">)</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'parent process:'</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">getppid</span><span class="p">())</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'process id:'</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">getpid</span><span class="p">())</span>
|
||
|
||
<span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
|
||
<span class="n">info</span><span class="p">(</span><span class="s1">'function f'</span><span class="p">)</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="n">info</span><span class="p">(</span><span class="s1">'main line'</span><span class="p">)</span>
|
||
<span class="n">p</span> <span class="o">=</span> <span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">f</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="s1">'bob'</span><span class="p">,))</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>For an explanation of why the <code class="docutils literal notranslate"><span class="pre">if</span> <span class="pre">__name__</span> <span class="pre">==</span> <span class="pre">'__main__'</span></code> part is
|
||
necessary, see <a class="reference internal" href="#multiprocessing-programming"><span class="std std-ref">Programming guidelines</span></a>.</p>
|
||
</div>
|
||
<div class="section" id="contexts-and-start-methods">
|
||
<h3>Contexts and start methods<a class="headerlink" href="#contexts-and-start-methods" title="Permalink to this headline">¶</a></h3>
|
||
<p id="multiprocessing-start-methods">Depending on the platform, <a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a> supports three ways
|
||
to start a process. These <em>start methods</em> are</p>
|
||
<blockquote>
|
||
<div><dl>
|
||
<dt><em>spawn</em></dt><dd><p>The parent process starts a fresh python interpreter process. The
|
||
child process will only inherit those resources necessary to run
|
||
the process objects <a class="reference internal" href="#multiprocessing.Process.run" title="multiprocessing.Process.run"><code class="xref py py-meth docutils literal notranslate"><span class="pre">run()</span></code></a> method. In particular,
|
||
unnecessary file descriptors and handles from the parent process
|
||
will not be inherited. Starting a process using this method is
|
||
rather slow compared to using <em>fork</em> or <em>forkserver</em>.</p>
|
||
<p>Available on Unix and Windows. The default on Windows.</p>
|
||
</dd>
|
||
<dt><em>fork</em></dt><dd><p>The parent process uses <a class="reference internal" href="os.html#os.fork" title="os.fork"><code class="xref py py-func docutils literal notranslate"><span class="pre">os.fork()</span></code></a> to fork the Python
|
||
interpreter. The child process, when it begins, is effectively
|
||
identical to the parent process. All resources of the parent are
|
||
inherited by the child process. Note that safely forking a
|
||
multithreaded process is problematic.</p>
|
||
<p>Available on Unix only. The default on Unix.</p>
|
||
</dd>
|
||
<dt><em>forkserver</em></dt><dd><p>When the program starts and selects the <em>forkserver</em> start method,
|
||
a server process is started. From then on, whenever a new process
|
||
is needed, the parent process connects to the server and requests
|
||
that it fork a new process. The fork server process is single
|
||
threaded so it is safe for it to use <a class="reference internal" href="os.html#os.fork" title="os.fork"><code class="xref py py-func docutils literal notranslate"><span class="pre">os.fork()</span></code></a>. No
|
||
unnecessary resources are inherited.</p>
|
||
<p>Available on Unix platforms which support passing file descriptors
|
||
over Unix pipes.</p>
|
||
</dd>
|
||
</dl>
|
||
</div></blockquote>
|
||
<div class="versionchanged">
|
||
<p><span class="versionmodified changed">Changed in version 3.4: </span><em>spawn</em> added on all unix platforms, and <em>forkserver</em> added for
|
||
some unix platforms.
|
||
Child processes no longer inherit all of the parents inheritable
|
||
handles on Windows.</p>
|
||
</div>
|
||
<p>On Unix using the <em>spawn</em> or <em>forkserver</em> start methods will also
|
||
start a <em>semaphore tracker</em> process which tracks the unlinked named
|
||
semaphores created by processes of the program. When all processes
|
||
have exited the semaphore tracker unlinks any remaining semaphores.
|
||
Usually there should be none, but if a process was killed by a signal
|
||
there may be some “leaked” semaphores. (Unlinking the named semaphores
|
||
is a serious matter since the system allows only a limited number, and
|
||
they will not be automatically unlinked until the next reboot.)</p>
|
||
<p>To select a start method you use the <a class="reference internal" href="#multiprocessing.set_start_method" title="multiprocessing.set_start_method"><code class="xref py py-func docutils literal notranslate"><span class="pre">set_start_method()</span></code></a> in
|
||
the <code class="docutils literal notranslate"><span class="pre">if</span> <span class="pre">__name__</span> <span class="pre">==</span> <span class="pre">'__main__'</span></code> clause of the main module. For
|
||
example:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">multiprocessing</span> <span class="k">as</span> <span class="nn">mp</span>
|
||
|
||
<span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="n">q</span><span class="p">):</span>
|
||
<span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">)</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="n">mp</span><span class="o">.</span><span class="n">set_start_method</span><span class="p">(</span><span class="s1">'spawn'</span><span class="p">)</span>
|
||
<span class="n">q</span> <span class="o">=</span> <span class="n">mp</span><span class="o">.</span><span class="n">Queue</span><span class="p">()</span>
|
||
<span class="n">p</span> <span class="o">=</span> <span class="n">mp</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">foo</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">q</span><span class="p">,))</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">())</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
|
||
</pre></div>
|
||
</div>
|
||
<p><a class="reference internal" href="#multiprocessing.set_start_method" title="multiprocessing.set_start_method"><code class="xref py py-func docutils literal notranslate"><span class="pre">set_start_method()</span></code></a> should not be used more than once in the
|
||
program.</p>
|
||
<p>Alternatively, you can use <a class="reference internal" href="#multiprocessing.get_context" title="multiprocessing.get_context"><code class="xref py py-func docutils literal notranslate"><span class="pre">get_context()</span></code></a> to obtain a context
|
||
object. Context objects have the same API as the multiprocessing
|
||
module, and allow one to use multiple start methods in the same
|
||
program.</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">multiprocessing</span> <span class="k">as</span> <span class="nn">mp</span>
|
||
|
||
<span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="n">q</span><span class="p">):</span>
|
||
<span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">)</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="n">ctx</span> <span class="o">=</span> <span class="n">mp</span><span class="o">.</span><span class="n">get_context</span><span class="p">(</span><span class="s1">'spawn'</span><span class="p">)</span>
|
||
<span class="n">q</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">Queue</span><span class="p">()</span>
|
||
<span class="n">p</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">foo</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">q</span><span class="p">,))</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">())</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>Note that objects related to one context may not be compatible with
|
||
processes for a different context. In particular, locks created using
|
||
the <em>fork</em> context cannot be passed to processes started using the
|
||
<em>spawn</em> or <em>forkserver</em> start methods.</p>
|
||
<p>A library which wants to use a particular start method should probably
|
||
use <a class="reference internal" href="#multiprocessing.get_context" title="multiprocessing.get_context"><code class="xref py py-func docutils literal notranslate"><span class="pre">get_context()</span></code></a> to avoid interfering with the choice of the
|
||
library user.</p>
|
||
<div class="admonition warning">
|
||
<p class="admonition-title">Warning</p>
|
||
<p>The <code class="docutils literal notranslate"><span class="pre">'spawn'</span></code> and <code class="docutils literal notranslate"><span class="pre">'forkserver'</span></code> start methods cannot currently
|
||
be used with “frozen” executables (i.e., binaries produced by
|
||
packages like <strong>PyInstaller</strong> and <strong>cx_Freeze</strong>) on Unix.
|
||
The <code class="docutils literal notranslate"><span class="pre">'fork'</span></code> start method does work.</p>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="exchanging-objects-between-processes">
|
||
<h3>Exchanging objects between processes<a class="headerlink" href="#exchanging-objects-between-processes" title="Permalink to this headline">¶</a></h3>
|
||
<p><a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a> supports two types of communication channel between
|
||
processes:</p>
|
||
<p><strong>Queues</strong></p>
|
||
<blockquote>
|
||
<div><p>The <a class="reference internal" href="#multiprocessing.Queue" title="multiprocessing.Queue"><code class="xref py py-class docutils literal notranslate"><span class="pre">Queue</span></code></a> class is a near clone of <a class="reference internal" href="queue.html#queue.Queue" title="queue.Queue"><code class="xref py py-class docutils literal notranslate"><span class="pre">queue.Queue</span></code></a>. For
|
||
example:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Process</span><span class="p">,</span> <span class="n">Queue</span>
|
||
|
||
<span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">q</span><span class="p">):</span>
|
||
<span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">([</span><span class="mi">42</span><span class="p">,</span> <span class="kc">None</span><span class="p">,</span> <span class="s1">'hello'</span><span class="p">])</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="n">q</span> <span class="o">=</span> <span class="n">Queue</span><span class="p">()</span>
|
||
<span class="n">p</span> <span class="o">=</span> <span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">f</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">q</span><span class="p">,))</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">())</span> <span class="c1"># prints "[42, None, 'hello']"</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>Queues are thread and process safe.</p>
|
||
</div></blockquote>
|
||
<p><strong>Pipes</strong></p>
|
||
<blockquote>
|
||
<div><p>The <a class="reference internal" href="#multiprocessing.Pipe" title="multiprocessing.Pipe"><code class="xref py py-func docutils literal notranslate"><span class="pre">Pipe()</span></code></a> function returns a pair of connection objects connected by a
|
||
pipe which by default is duplex (two-way). For example:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Process</span><span class="p">,</span> <span class="n">Pipe</span>
|
||
|
||
<span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">conn</span><span class="p">):</span>
|
||
<span class="n">conn</span><span class="o">.</span><span class="n">send</span><span class="p">([</span><span class="mi">42</span><span class="p">,</span> <span class="kc">None</span><span class="p">,</span> <span class="s1">'hello'</span><span class="p">])</span>
|
||
<span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="n">parent_conn</span><span class="p">,</span> <span class="n">child_conn</span> <span class="o">=</span> <span class="n">Pipe</span><span class="p">()</span>
|
||
<span class="n">p</span> <span class="o">=</span> <span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">f</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">child_conn</span><span class="p">,))</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">parent_conn</span><span class="o">.</span><span class="n">recv</span><span class="p">())</span> <span class="c1"># prints "[42, None, 'hello']"</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>The two connection objects returned by <a class="reference internal" href="#multiprocessing.Pipe" title="multiprocessing.Pipe"><code class="xref py py-func docutils literal notranslate"><span class="pre">Pipe()</span></code></a> represent the two ends of
|
||
the pipe. Each connection object has <code class="xref py py-meth docutils literal notranslate"><span class="pre">send()</span></code> and
|
||
<code class="xref py py-meth docutils literal notranslate"><span class="pre">recv()</span></code> methods (among others). Note that data in a pipe
|
||
may become corrupted if two processes (or threads) try to read from or write
|
||
to the <em>same</em> end of the pipe at the same time. Of course there is no risk
|
||
of corruption from processes using different ends of the pipe at the same
|
||
time.</p>
|
||
</div></blockquote>
|
||
</div>
|
||
<div class="section" id="synchronization-between-processes">
|
||
<h3>Synchronization between processes<a class="headerlink" href="#synchronization-between-processes" title="Permalink to this headline">¶</a></h3>
|
||
<p><a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a> contains equivalents of all the synchronization
|
||
primitives from <a class="reference internal" href="threading.html#module-threading" title="threading: Thread-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">threading</span></code></a>. For instance one can use a lock to ensure
|
||
that only one process prints to standard output at a time:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Process</span><span class="p">,</span> <span class="n">Lock</span>
|
||
|
||
<span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">l</span><span class="p">,</span> <span class="n">i</span><span class="p">):</span>
|
||
<span class="n">l</span><span class="o">.</span><span class="n">acquire</span><span class="p">()</span>
|
||
<span class="k">try</span><span class="p">:</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'hello world'</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span>
|
||
<span class="k">finally</span><span class="p">:</span>
|
||
<span class="n">l</span><span class="o">.</span><span class="n">release</span><span class="p">()</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="n">lock</span> <span class="o">=</span> <span class="n">Lock</span><span class="p">()</span>
|
||
|
||
<span class="k">for</span> <span class="n">num</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span>
|
||
<span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">f</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">lock</span><span class="p">,</span> <span class="n">num</span><span class="p">))</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>Without using the lock output from the different processes is liable to get all
|
||
mixed up.</p>
|
||
</div>
|
||
<div class="section" id="sharing-state-between-processes">
|
||
<h3>Sharing state between processes<a class="headerlink" href="#sharing-state-between-processes" title="Permalink to this headline">¶</a></h3>
|
||
<p>As mentioned above, when doing concurrent programming it is usually best to
|
||
avoid using shared state as far as possible. This is particularly true when
|
||
using multiple processes.</p>
|
||
<p>However, if you really do need to use some shared data then
|
||
<a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a> provides a couple of ways of doing so.</p>
|
||
<p><strong>Shared memory</strong></p>
|
||
<blockquote>
|
||
<div><p>Data can be stored in a shared memory map using <a class="reference internal" href="#multiprocessing.Value" title="multiprocessing.Value"><code class="xref py py-class docutils literal notranslate"><span class="pre">Value</span></code></a> or
|
||
<a class="reference internal" href="#multiprocessing.Array" title="multiprocessing.Array"><code class="xref py py-class docutils literal notranslate"><span class="pre">Array</span></code></a>. For example, the following code</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Process</span><span class="p">,</span> <span class="n">Value</span><span class="p">,</span> <span class="n">Array</span>
|
||
|
||
<span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">a</span><span class="p">):</span>
|
||
<span class="n">n</span><span class="o">.</span><span class="n">value</span> <span class="o">=</span> <span class="mf">3.1415927</span>
|
||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">a</span><span class="p">)):</span>
|
||
<span class="n">a</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="o">-</span><span class="n">a</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="n">num</span> <span class="o">=</span> <span class="n">Value</span><span class="p">(</span><span class="s1">'d'</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">)</span>
|
||
<span class="n">arr</span> <span class="o">=</span> <span class="n">Array</span><span class="p">(</span><span class="s1">'i'</span><span class="p">,</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">))</span>
|
||
|
||
<span class="n">p</span> <span class="o">=</span> <span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">f</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">num</span><span class="p">,</span> <span class="n">arr</span><span class="p">))</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
|
||
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">num</span><span class="o">.</span><span class="n">value</span><span class="p">)</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">arr</span><span class="p">[:])</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>will print</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="mf">3.1415927</span>
|
||
<span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="o">-</span><span class="mi">2</span><span class="p">,</span> <span class="o">-</span><span class="mi">3</span><span class="p">,</span> <span class="o">-</span><span class="mi">4</span><span class="p">,</span> <span class="o">-</span><span class="mi">5</span><span class="p">,</span> <span class="o">-</span><span class="mi">6</span><span class="p">,</span> <span class="o">-</span><span class="mi">7</span><span class="p">,</span> <span class="o">-</span><span class="mi">8</span><span class="p">,</span> <span class="o">-</span><span class="mi">9</span><span class="p">]</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>The <code class="docutils literal notranslate"><span class="pre">'d'</span></code> and <code class="docutils literal notranslate"><span class="pre">'i'</span></code> arguments used when creating <code class="docutils literal notranslate"><span class="pre">num</span></code> and <code class="docutils literal notranslate"><span class="pre">arr</span></code> are
|
||
typecodes of the kind used by the <a class="reference internal" href="array.html#module-array" title="array: Space efficient arrays of uniformly typed numeric values."><code class="xref py py-mod docutils literal notranslate"><span class="pre">array</span></code></a> module: <code class="docutils literal notranslate"><span class="pre">'d'</span></code> indicates a
|
||
double precision float and <code class="docutils literal notranslate"><span class="pre">'i'</span></code> indicates a signed integer. These shared
|
||
objects will be process and thread-safe.</p>
|
||
<p>For more flexibility in using shared memory one can use the
|
||
<a class="reference internal" href="#module-multiprocessing.sharedctypes" title="multiprocessing.sharedctypes: Allocate ctypes objects from shared memory."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing.sharedctypes</span></code></a> module which supports the creation of
|
||
arbitrary ctypes objects allocated from shared memory.</p>
|
||
</div></blockquote>
|
||
<p><strong>Server process</strong></p>
|
||
<blockquote>
|
||
<div><p>A manager object returned by <code class="xref py py-func docutils literal notranslate"><span class="pre">Manager()</span></code> controls a server process which
|
||
holds Python objects and allows other processes to manipulate them using
|
||
proxies.</p>
|
||
<p>A manager returned by <code class="xref py py-func docutils literal notranslate"><span class="pre">Manager()</span></code> will support types
|
||
<a class="reference internal" href="stdtypes.html#list" title="list"><code class="xref py py-class docutils literal notranslate"><span class="pre">list</span></code></a>, <a class="reference internal" href="stdtypes.html#dict" title="dict"><code class="xref py py-class docutils literal notranslate"><span class="pre">dict</span></code></a>, <a class="reference internal" href="#multiprocessing.managers.Namespace" title="multiprocessing.managers.Namespace"><code class="xref py py-class docutils literal notranslate"><span class="pre">Namespace</span></code></a>, <a class="reference internal" href="#multiprocessing.Lock" title="multiprocessing.Lock"><code class="xref py py-class docutils literal notranslate"><span class="pre">Lock</span></code></a>,
|
||
<a class="reference internal" href="#multiprocessing.RLock" title="multiprocessing.RLock"><code class="xref py py-class docutils literal notranslate"><span class="pre">RLock</span></code></a>, <a class="reference internal" href="#multiprocessing.Semaphore" title="multiprocessing.Semaphore"><code class="xref py py-class docutils literal notranslate"><span class="pre">Semaphore</span></code></a>, <a class="reference internal" href="#multiprocessing.BoundedSemaphore" title="multiprocessing.BoundedSemaphore"><code class="xref py py-class docutils literal notranslate"><span class="pre">BoundedSemaphore</span></code></a>,
|
||
<a class="reference internal" href="#multiprocessing.Condition" title="multiprocessing.Condition"><code class="xref py py-class docutils literal notranslate"><span class="pre">Condition</span></code></a>, <a class="reference internal" href="#multiprocessing.Event" title="multiprocessing.Event"><code class="xref py py-class docutils literal notranslate"><span class="pre">Event</span></code></a>, <a class="reference internal" href="#multiprocessing.Barrier" title="multiprocessing.Barrier"><code class="xref py py-class docutils literal notranslate"><span class="pre">Barrier</span></code></a>,
|
||
<a class="reference internal" href="#multiprocessing.Queue" title="multiprocessing.Queue"><code class="xref py py-class docutils literal notranslate"><span class="pre">Queue</span></code></a>, <a class="reference internal" href="#multiprocessing.Value" title="multiprocessing.Value"><code class="xref py py-class docutils literal notranslate"><span class="pre">Value</span></code></a> and <a class="reference internal" href="#multiprocessing.Array" title="multiprocessing.Array"><code class="xref py py-class docutils literal notranslate"><span class="pre">Array</span></code></a>. For example,</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Process</span><span class="p">,</span> <span class="n">Manager</span>
|
||
|
||
<span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">d</span><span class="p">,</span> <span class="n">l</span><span class="p">):</span>
|
||
<span class="n">d</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'1'</span>
|
||
<span class="n">d</span><span class="p">[</span><span class="s1">'2'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">2</span>
|
||
<span class="n">d</span><span class="p">[</span><span class="mf">0.25</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>
|
||
<span class="n">l</span><span class="o">.</span><span class="n">reverse</span><span class="p">()</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="k">with</span> <span class="n">Manager</span><span class="p">()</span> <span class="k">as</span> <span class="n">manager</span><span class="p">:</span>
|
||
<span class="n">d</span> <span class="o">=</span> <span class="n">manager</span><span class="o">.</span><span class="n">dict</span><span class="p">()</span>
|
||
<span class="n">l</span> <span class="o">=</span> <span class="n">manager</span><span class="o">.</span><span class="n">list</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">))</span>
|
||
|
||
<span class="n">p</span> <span class="o">=</span> <span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">f</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">d</span><span class="p">,</span> <span class="n">l</span><span class="p">))</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
|
||
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">d</span><span class="p">)</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">l</span><span class="p">)</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>will print</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="p">{</span><span class="mf">0.25</span><span class="p">:</span> <span class="kc">None</span><span class="p">,</span> <span class="mi">1</span><span class="p">:</span> <span class="s1">'1'</span><span class="p">,</span> <span class="s1">'2'</span><span class="p">:</span> <span class="mi">2</span><span class="p">}</span>
|
||
<span class="p">[</span><span class="mi">9</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">7</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">]</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>Server process managers are more flexible than using shared memory objects
|
||
because they can be made to support arbitrary object types. Also, a single
|
||
manager can be shared by processes on different computers over a network.
|
||
They are, however, slower than using shared memory.</p>
|
||
</div></blockquote>
|
||
</div>
|
||
<div class="section" id="using-a-pool-of-workers">
|
||
<h3>Using a pool of workers<a class="headerlink" href="#using-a-pool-of-workers" title="Permalink to this headline">¶</a></h3>
|
||
<p>The <a class="reference internal" href="#multiprocessing.pool.Pool" title="multiprocessing.pool.Pool"><code class="xref py py-class docutils literal notranslate"><span class="pre">Pool</span></code></a> class represents a pool of worker
|
||
processes. It has methods which allows tasks to be offloaded to the worker
|
||
processes in a few different ways.</p>
|
||
<p>For example:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Pool</span><span class="p">,</span> <span class="ne">TimeoutError</span>
|
||
<span class="kn">import</span> <span class="nn">time</span>
|
||
<span class="kn">import</span> <span class="nn">os</span>
|
||
|
||
<span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="n">x</span><span class="o">*</span><span class="n">x</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="c1"># start 4 worker processes</span>
|
||
<span class="k">with</span> <span class="n">Pool</span><span class="p">(</span><span class="n">processes</span><span class="o">=</span><span class="mi">4</span><span class="p">)</span> <span class="k">as</span> <span class="n">pool</span><span class="p">:</span>
|
||
|
||
<span class="c1"># print "[0, 1, 4,..., 81]"</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">pool</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)))</span>
|
||
|
||
<span class="c1"># print same numbers in arbitrary order</span>
|
||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">pool</span><span class="o">.</span><span class="n">imap_unordered</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)):</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">i</span><span class="p">)</span>
|
||
|
||
<span class="c1"># evaluate "f(20)" asynchronously</span>
|
||
<span class="n">res</span> <span class="o">=</span> <span class="n">pool</span><span class="o">.</span><span class="n">apply_async</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="p">(</span><span class="mi">20</span><span class="p">,))</span> <span class="c1"># runs in *only* one process</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">res</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">timeout</span><span class="o">=</span><span class="mi">1</span><span class="p">))</span> <span class="c1"># prints "400"</span>
|
||
|
||
<span class="c1"># evaluate "os.getpid()" asynchronously</span>
|
||
<span class="n">res</span> <span class="o">=</span> <span class="n">pool</span><span class="o">.</span><span class="n">apply_async</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getpid</span><span class="p">,</span> <span class="p">())</span> <span class="c1"># runs in *only* one process</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">res</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">timeout</span><span class="o">=</span><span class="mi">1</span><span class="p">))</span> <span class="c1"># prints the PID of that process</span>
|
||
|
||
<span class="c1"># launching multiple evaluations asynchronously *may* use more processes</span>
|
||
<span class="n">multiple_results</span> <span class="o">=</span> <span class="p">[</span><span class="n">pool</span><span class="o">.</span><span class="n">apply_async</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getpid</span><span class="p">,</span> <span class="p">())</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">4</span><span class="p">)]</span>
|
||
<span class="nb">print</span><span class="p">([</span><span class="n">res</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">timeout</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="k">for</span> <span class="n">res</span> <span class="ow">in</span> <span class="n">multiple_results</span><span class="p">])</span>
|
||
|
||
<span class="c1"># make a single worker sleep for 10 secs</span>
|
||
<span class="n">res</span> <span class="o">=</span> <span class="n">pool</span><span class="o">.</span><span class="n">apply_async</span><span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">,</span> <span class="p">(</span><span class="mi">10</span><span class="p">,))</span>
|
||
<span class="k">try</span><span class="p">:</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">res</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">timeout</span><span class="o">=</span><span class="mi">1</span><span class="p">))</span>
|
||
<span class="k">except</span> <span class="ne">TimeoutError</span><span class="p">:</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s2">"We lacked patience and got a multiprocessing.TimeoutError"</span><span class="p">)</span>
|
||
|
||
<span class="nb">print</span><span class="p">(</span><span class="s2">"For the moment, the pool remains available for more work"</span><span class="p">)</span>
|
||
|
||
<span class="c1"># exiting the 'with'-block has stopped the pool</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Now the pool is closed and no longer available"</span><span class="p">)</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>Note that the methods of a pool should only ever be used by the
|
||
process which created it.</p>
|
||
<div class="admonition note">
|
||
<p class="admonition-title">Note</p>
|
||
<p>Functionality within this package requires that the <code class="docutils literal notranslate"><span class="pre">__main__</span></code> module be
|
||
importable by the children. This is covered in <a class="reference internal" href="#multiprocessing-programming"><span class="std std-ref">Programming guidelines</span></a>
|
||
however it is worth pointing out here. This means that some examples, such
|
||
as the <a class="reference internal" href="#multiprocessing.pool.Pool" title="multiprocessing.pool.Pool"><code class="xref py py-class docutils literal notranslate"><span class="pre">multiprocessing.pool.Pool</span></code></a> examples will not work in the
|
||
interactive interpreter. For example:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Pool</span>
|
||
<span class="gp">>>> </span><span class="n">p</span> <span class="o">=</span> <span class="n">Pool</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
|
||
<span class="gp">>>> </span><span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
|
||
<span class="gp">... </span> <span class="k">return</span> <span class="n">x</span><span class="o">*</span><span class="n">x</span>
|
||
<span class="gp">...</span>
|
||
<span class="gp">>>> </span><span class="n">p</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">])</span>
|
||
<span class="go">Process PoolWorker-1:</span>
|
||
<span class="go">Process PoolWorker-2:</span>
|
||
<span class="go">Process PoolWorker-3:</span>
|
||
<span class="gt">Traceback (most recent call last):</span>
|
||
<span class="gr">AttributeError</span>: <span class="n">'module' object has no attribute 'f'</span>
|
||
<span class="go">AttributeError: 'module' object has no attribute 'f'</span>
|
||
<span class="go">AttributeError: 'module' object has no attribute 'f'</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>(If you try this it will actually output three full tracebacks
|
||
interleaved in a semi-random fashion, and then you may have to
|
||
stop the master process somehow.)</p>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="reference">
|
||
<h2>Reference<a class="headerlink" href="#reference" title="Permalink to this headline">¶</a></h2>
|
||
<p>The <a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a> package mostly replicates the API of the
|
||
<a class="reference internal" href="threading.html#module-threading" title="threading: Thread-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">threading</span></code></a> module.</p>
|
||
<div class="section" id="process-and-exceptions">
|
||
<h3><a class="reference internal" href="#multiprocessing.Process" title="multiprocessing.Process"><code class="xref py py-class docutils literal notranslate"><span class="pre">Process</span></code></a> and exceptions<a class="headerlink" href="#process-and-exceptions" title="Permalink to this headline">¶</a></h3>
|
||
<dl class="class">
|
||
<dt id="multiprocessing.Process">
|
||
<em class="property">class </em><code class="descclassname">multiprocessing.</code><code class="descname">Process</code><span class="sig-paren">(</span><em>group=None</em>, <em>target=None</em>, <em>name=None</em>, <em>args=()</em>, <em>kwargs={}</em>, <em>*</em>, <em>daemon=None</em><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Process" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Process objects represent activity that is run in a separate process. The
|
||
<a class="reference internal" href="#multiprocessing.Process" title="multiprocessing.Process"><code class="xref py py-class docutils literal notranslate"><span class="pre">Process</span></code></a> class has equivalents of all the methods of
|
||
<a class="reference internal" href="threading.html#threading.Thread" title="threading.Thread"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.Thread</span></code></a>.</p>
|
||
<p>The constructor should always be called with keyword arguments. <em>group</em>
|
||
should always be <code class="docutils literal notranslate"><span class="pre">None</span></code>; it exists solely for compatibility with
|
||
<a class="reference internal" href="threading.html#threading.Thread" title="threading.Thread"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.Thread</span></code></a>. <em>target</em> is the callable object to be invoked by
|
||
the <a class="reference internal" href="#multiprocessing.Process.run" title="multiprocessing.Process.run"><code class="xref py py-meth docutils literal notranslate"><span class="pre">run()</span></code></a> method. It defaults to <code class="docutils literal notranslate"><span class="pre">None</span></code>, meaning nothing is
|
||
called. <em>name</em> is the process name (see <a class="reference internal" href="#multiprocessing.Process.name" title="multiprocessing.Process.name"><code class="xref py py-attr docutils literal notranslate"><span class="pre">name</span></code></a> for more details).
|
||
<em>args</em> is the argument tuple for the target invocation. <em>kwargs</em> is a
|
||
dictionary of keyword arguments for the target invocation. If provided,
|
||
the keyword-only <em>daemon</em> argument sets the process <a class="reference internal" href="#multiprocessing.Process.daemon" title="multiprocessing.Process.daemon"><code class="xref py py-attr docutils literal notranslate"><span class="pre">daemon</span></code></a> flag
|
||
to <code class="docutils literal notranslate"><span class="pre">True</span></code> or <code class="docutils literal notranslate"><span class="pre">False</span></code>. If <code class="docutils literal notranslate"><span class="pre">None</span></code> (the default), this flag will be
|
||
inherited from the creating process.</p>
|
||
<p>By default, no arguments are passed to <em>target</em>.</p>
|
||
<p>If a subclass overrides the constructor, it must make sure it invokes the
|
||
base class constructor (<code class="xref py py-meth docutils literal notranslate"><span class="pre">Process.__init__()</span></code>) before doing anything else
|
||
to the process.</p>
|
||
<div class="versionchanged">
|
||
<p><span class="versionmodified changed">Changed in version 3.3: </span>Added the <em>daemon</em> argument.</p>
|
||
</div>
|
||
<dl class="method">
|
||
<dt id="multiprocessing.Process.run">
|
||
<code class="descname">run</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Process.run" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Method representing the process’s activity.</p>
|
||
<p>You may override this method in a subclass. The standard <a class="reference internal" href="#multiprocessing.Process.run" title="multiprocessing.Process.run"><code class="xref py py-meth docutils literal notranslate"><span class="pre">run()</span></code></a>
|
||
method invokes the callable object passed to the object’s constructor as
|
||
the target argument, if any, with sequential and keyword arguments taken
|
||
from the <em>args</em> and <em>kwargs</em> arguments, respectively.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.Process.start">
|
||
<code class="descname">start</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Process.start" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Start the process’s activity.</p>
|
||
<p>This must be called at most once per process object. It arranges for the
|
||
object’s <a class="reference internal" href="#multiprocessing.Process.run" title="multiprocessing.Process.run"><code class="xref py py-meth docutils literal notranslate"><span class="pre">run()</span></code></a> method to be invoked in a separate process.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.Process.join">
|
||
<code class="descname">join</code><span class="sig-paren">(</span><span class="optional">[</span><em>timeout</em><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Process.join" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>If the optional argument <em>timeout</em> is <code class="docutils literal notranslate"><span class="pre">None</span></code> (the default), the method
|
||
blocks until the process whose <a class="reference internal" href="#multiprocessing.Process.join" title="multiprocessing.Process.join"><code class="xref py py-meth docutils literal notranslate"><span class="pre">join()</span></code></a> method is called terminates.
|
||
If <em>timeout</em> is a positive number, it blocks at most <em>timeout</em> seconds.
|
||
Note that the method returns <code class="docutils literal notranslate"><span class="pre">None</span></code> if its process terminates or if the
|
||
method times out. Check the process’s <a class="reference internal" href="#multiprocessing.Process.exitcode" title="multiprocessing.Process.exitcode"><code class="xref py py-attr docutils literal notranslate"><span class="pre">exitcode</span></code></a> to determine if
|
||
it terminated.</p>
|
||
<p>A process can be joined many times.</p>
|
||
<p>A process cannot join itself because this would cause a deadlock. It is
|
||
an error to attempt to join a process before it has been started.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="attribute">
|
||
<dt id="multiprocessing.Process.name">
|
||
<code class="descname">name</code><a class="headerlink" href="#multiprocessing.Process.name" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>The process’s name. The name is a string used for identification purposes
|
||
only. It has no semantics. Multiple processes may be given the same
|
||
name.</p>
|
||
<p>The initial name is set by the constructor. If no explicit name is
|
||
provided to the constructor, a name of the form
|
||
‘Process-N<sub>1</sub>:N<sub>2</sub>:…:N<sub>k</sub>’ is constructed, where
|
||
each N<sub>k</sub> is the N-th child of its parent.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.Process.is_alive">
|
||
<code class="descname">is_alive</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Process.is_alive" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return whether the process is alive.</p>
|
||
<p>Roughly, a process object is alive from the moment the <a class="reference internal" href="#multiprocessing.Process.start" title="multiprocessing.Process.start"><code class="xref py py-meth docutils literal notranslate"><span class="pre">start()</span></code></a>
|
||
method returns until the child process terminates.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="attribute">
|
||
<dt id="multiprocessing.Process.daemon">
|
||
<code class="descname">daemon</code><a class="headerlink" href="#multiprocessing.Process.daemon" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>The process’s daemon flag, a Boolean value. This must be set before
|
||
<a class="reference internal" href="#multiprocessing.Process.start" title="multiprocessing.Process.start"><code class="xref py py-meth docutils literal notranslate"><span class="pre">start()</span></code></a> is called.</p>
|
||
<p>The initial value is inherited from the creating process.</p>
|
||
<p>When a process exits, it attempts to terminate all of its daemonic child
|
||
processes.</p>
|
||
<p>Note that a daemonic process is not allowed to create child processes.
|
||
Otherwise a daemonic process would leave its children orphaned if it gets
|
||
terminated when its parent process exits. Additionally, these are <strong>not</strong>
|
||
Unix daemons or services, they are normal processes that will be
|
||
terminated (and not joined) if non-daemonic processes have exited.</p>
|
||
</dd></dl>
|
||
|
||
<p>In addition to the <a class="reference internal" href="threading.html#threading.Thread" title="threading.Thread"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.Thread</span></code></a> API, <a class="reference internal" href="#multiprocessing.Process" title="multiprocessing.Process"><code class="xref py py-class docutils literal notranslate"><span class="pre">Process</span></code></a> objects
|
||
also support the following attributes and methods:</p>
|
||
<dl class="attribute">
|
||
<dt id="multiprocessing.Process.pid">
|
||
<code class="descname">pid</code><a class="headerlink" href="#multiprocessing.Process.pid" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return the process ID. Before the process is spawned, this will be
|
||
<code class="docutils literal notranslate"><span class="pre">None</span></code>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="attribute">
|
||
<dt id="multiprocessing.Process.exitcode">
|
||
<code class="descname">exitcode</code><a class="headerlink" href="#multiprocessing.Process.exitcode" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>The child’s exit code. This will be <code class="docutils literal notranslate"><span class="pre">None</span></code> if the process has not yet
|
||
terminated. A negative value <em>-N</em> indicates that the child was terminated
|
||
by signal <em>N</em>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="attribute">
|
||
<dt id="multiprocessing.Process.authkey">
|
||
<code class="descname">authkey</code><a class="headerlink" href="#multiprocessing.Process.authkey" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>The process’s authentication key (a byte string).</p>
|
||
<p>When <a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a> is initialized the main process is assigned a
|
||
random string using <a class="reference internal" href="os.html#os.urandom" title="os.urandom"><code class="xref py py-func docutils literal notranslate"><span class="pre">os.urandom()</span></code></a>.</p>
|
||
<p>When a <a class="reference internal" href="#multiprocessing.Process" title="multiprocessing.Process"><code class="xref py py-class docutils literal notranslate"><span class="pre">Process</span></code></a> object is created, it will inherit the
|
||
authentication key of its parent process, although this may be changed by
|
||
setting <a class="reference internal" href="#multiprocessing.Process.authkey" title="multiprocessing.Process.authkey"><code class="xref py py-attr docutils literal notranslate"><span class="pre">authkey</span></code></a> to another byte string.</p>
|
||
<p>See <a class="reference internal" href="#multiprocessing-auth-keys"><span class="std std-ref">Authentication keys</span></a>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="attribute">
|
||
<dt id="multiprocessing.Process.sentinel">
|
||
<code class="descname">sentinel</code><a class="headerlink" href="#multiprocessing.Process.sentinel" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>A numeric handle of a system object which will become “ready” when
|
||
the process ends.</p>
|
||
<p>You can use this value if you want to wait on several events at
|
||
once using <a class="reference internal" href="#multiprocessing.connection.wait" title="multiprocessing.connection.wait"><code class="xref py py-func docutils literal notranslate"><span class="pre">multiprocessing.connection.wait()</span></code></a>. Otherwise
|
||
calling <a class="reference internal" href="#multiprocessing.Process.join" title="multiprocessing.Process.join"><code class="xref py py-meth docutils literal notranslate"><span class="pre">join()</span></code></a> is simpler.</p>
|
||
<p>On Windows, this is an OS handle usable with the <code class="docutils literal notranslate"><span class="pre">WaitForSingleObject</span></code>
|
||
and <code class="docutils literal notranslate"><span class="pre">WaitForMultipleObjects</span></code> family of API calls. On Unix, this is
|
||
a file descriptor usable with primitives from the <a class="reference internal" href="select.html#module-select" title="select: Wait for I/O completion on multiple streams."><code class="xref py py-mod docutils literal notranslate"><span class="pre">select</span></code></a> module.</p>
|
||
<div class="versionadded">
|
||
<p><span class="versionmodified added">New in version 3.3.</span></p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.Process.terminate">
|
||
<code class="descname">terminate</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Process.terminate" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Terminate the process. On Unix this is done using the <code class="docutils literal notranslate"><span class="pre">SIGTERM</span></code> signal;
|
||
on Windows <code class="xref c c-func docutils literal notranslate"><span class="pre">TerminateProcess()</span></code> is used. Note that exit handlers and
|
||
finally clauses, etc., will not be executed.</p>
|
||
<p>Note that descendant processes of the process will <em>not</em> be terminated –
|
||
they will simply become orphaned.</p>
|
||
<div class="admonition warning">
|
||
<p class="admonition-title">Warning</p>
|
||
<p>If this method is used when the associated process is using a pipe or
|
||
queue then the pipe or queue is liable to become corrupted and may
|
||
become unusable by other process. Similarly, if the process has
|
||
acquired a lock or semaphore etc. then terminating it is liable to
|
||
cause other processes to deadlock.</p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.Process.kill">
|
||
<code class="descname">kill</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Process.kill" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Same as <a class="reference internal" href="#multiprocessing.Process.terminate" title="multiprocessing.Process.terminate"><code class="xref py py-meth docutils literal notranslate"><span class="pre">terminate()</span></code></a> but using the <code class="docutils literal notranslate"><span class="pre">SIGKILL</span></code> signal on Unix.</p>
|
||
<div class="versionadded">
|
||
<p><span class="versionmodified added">New in version 3.7.</span></p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.Process.close">
|
||
<code class="descname">close</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Process.close" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Close the <a class="reference internal" href="#multiprocessing.Process" title="multiprocessing.Process"><code class="xref py py-class docutils literal notranslate"><span class="pre">Process</span></code></a> object, releasing all resources associated
|
||
with it. <a class="reference internal" href="exceptions.html#ValueError" title="ValueError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">ValueError</span></code></a> is raised if the underlying process
|
||
is still running. Once <a class="reference internal" href="#multiprocessing.Process.close" title="multiprocessing.Process.close"><code class="xref py py-meth docutils literal notranslate"><span class="pre">close()</span></code></a> returns successfully, most
|
||
other methods and attributes of the <a class="reference internal" href="#multiprocessing.Process" title="multiprocessing.Process"><code class="xref py py-class docutils literal notranslate"><span class="pre">Process</span></code></a> object will
|
||
raise <a class="reference internal" href="exceptions.html#ValueError" title="ValueError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">ValueError</span></code></a>.</p>
|
||
<div class="versionadded">
|
||
<p><span class="versionmodified added">New in version 3.7.</span></p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<p>Note that the <a class="reference internal" href="#multiprocessing.Process.start" title="multiprocessing.Process.start"><code class="xref py py-meth docutils literal notranslate"><span class="pre">start()</span></code></a>, <a class="reference internal" href="#multiprocessing.Process.join" title="multiprocessing.Process.join"><code class="xref py py-meth docutils literal notranslate"><span class="pre">join()</span></code></a>, <a class="reference internal" href="#multiprocessing.Process.is_alive" title="multiprocessing.Process.is_alive"><code class="xref py py-meth docutils literal notranslate"><span class="pre">is_alive()</span></code></a>,
|
||
<a class="reference internal" href="#multiprocessing.Process.terminate" title="multiprocessing.Process.terminate"><code class="xref py py-meth docutils literal notranslate"><span class="pre">terminate()</span></code></a> and <a class="reference internal" href="#multiprocessing.Process.exitcode" title="multiprocessing.Process.exitcode"><code class="xref py py-attr docutils literal notranslate"><span class="pre">exitcode</span></code></a> methods should only be called by
|
||
the process that created the process object.</p>
|
||
<p>Example usage of some of the methods of <a class="reference internal" href="#multiprocessing.Process" title="multiprocessing.Process"><code class="xref py py-class docutils literal notranslate"><span class="pre">Process</span></code></a>:</p>
|
||
<div class="highlight-pycon3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">multiprocessing</span><span class="o">,</span> <span class="nn">time</span><span class="o">,</span> <span class="nn">signal</span>
|
||
<span class="gp">>>> </span><span class="n">p</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="mi">1000</span><span class="p">,))</span>
|
||
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">is_alive</span><span class="p">())</span>
|
||
<span class="go"><Process(Process-1, initial)> False</span>
|
||
<span class="gp">>>> </span><span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
|
||
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">is_alive</span><span class="p">())</span>
|
||
<span class="go"><Process(Process-1, started)> True</span>
|
||
<span class="gp">>>> </span><span class="n">p</span><span class="o">.</span><span class="n">terminate</span><span class="p">()</span>
|
||
<span class="gp">>>> </span><span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span>
|
||
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">is_alive</span><span class="p">())</span>
|
||
<span class="go"><Process(Process-1, stopped[SIGTERM])> False</span>
|
||
<span class="gp">>>> </span><span class="n">p</span><span class="o">.</span><span class="n">exitcode</span> <span class="o">==</span> <span class="o">-</span><span class="n">signal</span><span class="o">.</span><span class="n">SIGTERM</span>
|
||
<span class="go">True</span>
|
||
</pre></div>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="exception">
|
||
<dt id="multiprocessing.ProcessError">
|
||
<em class="property">exception </em><code class="descclassname">multiprocessing.</code><code class="descname">ProcessError</code><a class="headerlink" href="#multiprocessing.ProcessError" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>The base class of all <a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a> exceptions.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="exception">
|
||
<dt id="multiprocessing.BufferTooShort">
|
||
<em class="property">exception </em><code class="descclassname">multiprocessing.</code><code class="descname">BufferTooShort</code><a class="headerlink" href="#multiprocessing.BufferTooShort" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Exception raised by <code class="xref py py-meth docutils literal notranslate"><span class="pre">Connection.recv_bytes_into()</span></code> when the supplied
|
||
buffer object is too small for the message read.</p>
|
||
<p>If <code class="docutils literal notranslate"><span class="pre">e</span></code> is an instance of <a class="reference internal" href="#multiprocessing.BufferTooShort" title="multiprocessing.BufferTooShort"><code class="xref py py-exc docutils literal notranslate"><span class="pre">BufferTooShort</span></code></a> then <code class="docutils literal notranslate"><span class="pre">e.args[0]</span></code> will give
|
||
the message as a byte string.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="exception">
|
||
<dt id="multiprocessing.AuthenticationError">
|
||
<em class="property">exception </em><code class="descclassname">multiprocessing.</code><code class="descname">AuthenticationError</code><a class="headerlink" href="#multiprocessing.AuthenticationError" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Raised when there is an authentication error.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="exception">
|
||
<dt id="multiprocessing.TimeoutError">
|
||
<em class="property">exception </em><code class="descclassname">multiprocessing.</code><code class="descname">TimeoutError</code><a class="headerlink" href="#multiprocessing.TimeoutError" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Raised by methods with a timeout when the timeout expires.</p>
|
||
</dd></dl>
|
||
|
||
</div>
|
||
<div class="section" id="pipes-and-queues">
|
||
<h3>Pipes and Queues<a class="headerlink" href="#pipes-and-queues" title="Permalink to this headline">¶</a></h3>
|
||
<p>When using multiple processes, one generally uses message passing for
|
||
communication between processes and avoids having to use any synchronization
|
||
primitives like locks.</p>
|
||
<p>For passing messages one can use <a class="reference internal" href="#multiprocessing.Pipe" title="multiprocessing.Pipe"><code class="xref py py-func docutils literal notranslate"><span class="pre">Pipe()</span></code></a> (for a connection between two
|
||
processes) or a queue (which allows multiple producers and consumers).</p>
|
||
<p>The <a class="reference internal" href="#multiprocessing.Queue" title="multiprocessing.Queue"><code class="xref py py-class docutils literal notranslate"><span class="pre">Queue</span></code></a>, <a class="reference internal" href="#multiprocessing.SimpleQueue" title="multiprocessing.SimpleQueue"><code class="xref py py-class docutils literal notranslate"><span class="pre">SimpleQueue</span></code></a> and <a class="reference internal" href="#multiprocessing.JoinableQueue" title="multiprocessing.JoinableQueue"><code class="xref py py-class docutils literal notranslate"><span class="pre">JoinableQueue</span></code></a> types
|
||
are multi-producer, multi-consumer <abbr title="first-in, first-out">FIFO</abbr>
|
||
queues modelled on the <a class="reference internal" href="queue.html#queue.Queue" title="queue.Queue"><code class="xref py py-class docutils literal notranslate"><span class="pre">queue.Queue</span></code></a> class in the
|
||
standard library. They differ in that <a class="reference internal" href="#multiprocessing.Queue" title="multiprocessing.Queue"><code class="xref py py-class docutils literal notranslate"><span class="pre">Queue</span></code></a> lacks the
|
||
<a class="reference internal" href="queue.html#queue.Queue.task_done" title="queue.Queue.task_done"><code class="xref py py-meth docutils literal notranslate"><span class="pre">task_done()</span></code></a> and <a class="reference internal" href="queue.html#queue.Queue.join" title="queue.Queue.join"><code class="xref py py-meth docutils literal notranslate"><span class="pre">join()</span></code></a> methods introduced
|
||
into Python 2.5’s <a class="reference internal" href="queue.html#queue.Queue" title="queue.Queue"><code class="xref py py-class docutils literal notranslate"><span class="pre">queue.Queue</span></code></a> class.</p>
|
||
<p>If you use <a class="reference internal" href="#multiprocessing.JoinableQueue" title="multiprocessing.JoinableQueue"><code class="xref py py-class docutils literal notranslate"><span class="pre">JoinableQueue</span></code></a> then you <strong>must</strong> call
|
||
<a class="reference internal" href="#multiprocessing.JoinableQueue.task_done" title="multiprocessing.JoinableQueue.task_done"><code class="xref py py-meth docutils literal notranslate"><span class="pre">JoinableQueue.task_done()</span></code></a> for each task removed from the queue or else the
|
||
semaphore used to count the number of unfinished tasks may eventually overflow,
|
||
raising an exception.</p>
|
||
<p>Note that one can also create a shared queue by using a manager object – see
|
||
<a class="reference internal" href="#multiprocessing-managers"><span class="std std-ref">Managers</span></a>.</p>
|
||
<div class="admonition note">
|
||
<p class="admonition-title">Note</p>
|
||
<p><a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a> uses the usual <a class="reference internal" href="queue.html#queue.Empty" title="queue.Empty"><code class="xref py py-exc docutils literal notranslate"><span class="pre">queue.Empty</span></code></a> and
|
||
<a class="reference internal" href="queue.html#queue.Full" title="queue.Full"><code class="xref py py-exc docutils literal notranslate"><span class="pre">queue.Full</span></code></a> exceptions to signal a timeout. They are not available in
|
||
the <a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a> namespace so you need to import them from
|
||
<a class="reference internal" href="queue.html#module-queue" title="queue: A synchronized queue class."><code class="xref py py-mod docutils literal notranslate"><span class="pre">queue</span></code></a>.</p>
|
||
</div>
|
||
<div class="admonition note">
|
||
<p class="admonition-title">Note</p>
|
||
<p>When an object is put on a queue, the object is pickled and a
|
||
background thread later flushes the pickled data to an underlying
|
||
pipe. This has some consequences which are a little surprising,
|
||
but should not cause any practical difficulties – if they really
|
||
bother you then you can instead use a queue created with a
|
||
<a class="reference internal" href="#multiprocessing-managers"><span class="std std-ref">manager</span></a>.</p>
|
||
<ol class="arabic simple">
|
||
<li><p>After putting an object on an empty queue there may be an
|
||
infinitesimal delay before the queue’s <a class="reference internal" href="#multiprocessing.Queue.empty" title="multiprocessing.Queue.empty"><code class="xref py py-meth docutils literal notranslate"><span class="pre">empty()</span></code></a>
|
||
method returns <a class="reference internal" href="constants.html#False" title="False"><code class="xref py py-const docutils literal notranslate"><span class="pre">False</span></code></a> and <a class="reference internal" href="#multiprocessing.Queue.get_nowait" title="multiprocessing.Queue.get_nowait"><code class="xref py py-meth docutils literal notranslate"><span class="pre">get_nowait()</span></code></a> can
|
||
return without raising <a class="reference internal" href="queue.html#queue.Empty" title="queue.Empty"><code class="xref py py-exc docutils literal notranslate"><span class="pre">queue.Empty</span></code></a>.</p></li>
|
||
<li><p>If multiple processes are enqueuing objects, it is possible for
|
||
the objects to be received at the other end out-of-order.
|
||
However, objects enqueued by the same process will always be in
|
||
the expected order with respect to each other.</p></li>
|
||
</ol>
|
||
</div>
|
||
<div class="admonition warning">
|
||
<p class="admonition-title">Warning</p>
|
||
<p>If a process is killed using <a class="reference internal" href="#multiprocessing.Process.terminate" title="multiprocessing.Process.terminate"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Process.terminate()</span></code></a> or <a class="reference internal" href="os.html#os.kill" title="os.kill"><code class="xref py py-func docutils literal notranslate"><span class="pre">os.kill()</span></code></a>
|
||
while it is trying to use a <a class="reference internal" href="#multiprocessing.Queue" title="multiprocessing.Queue"><code class="xref py py-class docutils literal notranslate"><span class="pre">Queue</span></code></a>, then the data in the queue is
|
||
likely to become corrupted. This may cause any other process to get an
|
||
exception when it tries to use the queue later on.</p>
|
||
</div>
|
||
<div class="admonition warning">
|
||
<p class="admonition-title">Warning</p>
|
||
<p>As mentioned above, if a child process has put items on a queue (and it has
|
||
not used <a class="reference internal" href="#multiprocessing.Queue.cancel_join_thread" title="multiprocessing.Queue.cancel_join_thread"><code class="xref py py-meth docutils literal notranslate"><span class="pre">JoinableQueue.cancel_join_thread</span></code></a>), then that process will
|
||
not terminate until all buffered items have been flushed to the pipe.</p>
|
||
<p>This means that if you try joining that process you may get a deadlock unless
|
||
you are sure that all items which have been put on the queue have been
|
||
consumed. Similarly, if the child process is non-daemonic then the parent
|
||
process may hang on exit when it tries to join all its non-daemonic children.</p>
|
||
<p>Note that a queue created using a manager does not have this issue. See
|
||
<a class="reference internal" href="#multiprocessing-programming"><span class="std std-ref">Programming guidelines</span></a>.</p>
|
||
</div>
|
||
<p>For an example of the usage of queues for interprocess communication see
|
||
<a class="reference internal" href="#multiprocessing-examples"><span class="std std-ref">Examples</span></a>.</p>
|
||
<dl class="function">
|
||
<dt id="multiprocessing.Pipe">
|
||
<code class="descclassname">multiprocessing.</code><code class="descname">Pipe</code><span class="sig-paren">(</span><span class="optional">[</span><em>duplex</em><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Pipe" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Returns a pair <code class="docutils literal notranslate"><span class="pre">(conn1,</span> <span class="pre">conn2)</span></code> of
|
||
<a class="reference internal" href="#multiprocessing.connection.Connection" title="multiprocessing.connection.Connection"><code class="xref py py-class docutils literal notranslate"><span class="pre">Connection</span></code></a> objects representing the
|
||
ends of a pipe.</p>
|
||
<p>If <em>duplex</em> is <code class="docutils literal notranslate"><span class="pre">True</span></code> (the default) then the pipe is bidirectional. If
|
||
<em>duplex</em> is <code class="docutils literal notranslate"><span class="pre">False</span></code> then the pipe is unidirectional: <code class="docutils literal notranslate"><span class="pre">conn1</span></code> can only be
|
||
used for receiving messages and <code class="docutils literal notranslate"><span class="pre">conn2</span></code> can only be used for sending
|
||
messages.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="class">
|
||
<dt id="multiprocessing.Queue">
|
||
<em class="property">class </em><code class="descclassname">multiprocessing.</code><code class="descname">Queue</code><span class="sig-paren">(</span><span class="optional">[</span><em>maxsize</em><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Queue" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Returns a process shared queue implemented using a pipe and a few
|
||
locks/semaphores. When a process first puts an item on the queue a feeder
|
||
thread is started which transfers objects from a buffer into the pipe.</p>
|
||
<p>The usual <a class="reference internal" href="queue.html#queue.Empty" title="queue.Empty"><code class="xref py py-exc docutils literal notranslate"><span class="pre">queue.Empty</span></code></a> and <a class="reference internal" href="queue.html#queue.Full" title="queue.Full"><code class="xref py py-exc docutils literal notranslate"><span class="pre">queue.Full</span></code></a> exceptions from the
|
||
standard library’s <a class="reference internal" href="queue.html#module-queue" title="queue: A synchronized queue class."><code class="xref py py-mod docutils literal notranslate"><span class="pre">queue</span></code></a> module are raised to signal timeouts.</p>
|
||
<p><a class="reference internal" href="#multiprocessing.Queue" title="multiprocessing.Queue"><code class="xref py py-class docutils literal notranslate"><span class="pre">Queue</span></code></a> implements all the methods of <a class="reference internal" href="queue.html#queue.Queue" title="queue.Queue"><code class="xref py py-class docutils literal notranslate"><span class="pre">queue.Queue</span></code></a> except for
|
||
<a class="reference internal" href="queue.html#queue.Queue.task_done" title="queue.Queue.task_done"><code class="xref py py-meth docutils literal notranslate"><span class="pre">task_done()</span></code></a> and <a class="reference internal" href="queue.html#queue.Queue.join" title="queue.Queue.join"><code class="xref py py-meth docutils literal notranslate"><span class="pre">join()</span></code></a>.</p>
|
||
<dl class="method">
|
||
<dt id="multiprocessing.Queue.qsize">
|
||
<code class="descname">qsize</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Queue.qsize" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return the approximate size of the queue. Because of
|
||
multithreading/multiprocessing semantics, this number is not reliable.</p>
|
||
<p>Note that this may raise <a class="reference internal" href="exceptions.html#NotImplementedError" title="NotImplementedError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">NotImplementedError</span></code></a> on Unix platforms like
|
||
Mac OS X where <code class="docutils literal notranslate"><span class="pre">sem_getvalue()</span></code> is not implemented.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.Queue.empty">
|
||
<code class="descname">empty</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Queue.empty" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return <code class="docutils literal notranslate"><span class="pre">True</span></code> if the queue is empty, <code class="docutils literal notranslate"><span class="pre">False</span></code> otherwise. Because of
|
||
multithreading/multiprocessing semantics, this is not reliable.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.Queue.full">
|
||
<code class="descname">full</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Queue.full" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return <code class="docutils literal notranslate"><span class="pre">True</span></code> if the queue is full, <code class="docutils literal notranslate"><span class="pre">False</span></code> otherwise. Because of
|
||
multithreading/multiprocessing semantics, this is not reliable.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.Queue.put">
|
||
<code class="descname">put</code><span class="sig-paren">(</span><em>obj</em><span class="optional">[</span>, <em>block</em><span class="optional">[</span>, <em>timeout</em><span class="optional">]</span><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Queue.put" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Put obj into the queue. If the optional argument <em>block</em> is <code class="docutils literal notranslate"><span class="pre">True</span></code>
|
||
(the default) and <em>timeout</em> is <code class="docutils literal notranslate"><span class="pre">None</span></code> (the default), block if necessary until
|
||
a free slot is available. If <em>timeout</em> is a positive number, it blocks at
|
||
most <em>timeout</em> seconds and raises the <a class="reference internal" href="queue.html#queue.Full" title="queue.Full"><code class="xref py py-exc docutils literal notranslate"><span class="pre">queue.Full</span></code></a> exception if no
|
||
free slot was available within that time. Otherwise (<em>block</em> is
|
||
<code class="docutils literal notranslate"><span class="pre">False</span></code>), put an item on the queue if a free slot is immediately
|
||
available, else raise the <a class="reference internal" href="queue.html#queue.Full" title="queue.Full"><code class="xref py py-exc docutils literal notranslate"><span class="pre">queue.Full</span></code></a> exception (<em>timeout</em> is
|
||
ignored in that case).</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.Queue.put_nowait">
|
||
<code class="descname">put_nowait</code><span class="sig-paren">(</span><em>obj</em><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Queue.put_nowait" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Equivalent to <code class="docutils literal notranslate"><span class="pre">put(obj,</span> <span class="pre">False)</span></code>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.Queue.get">
|
||
<code class="descname">get</code><span class="sig-paren">(</span><span class="optional">[</span><em>block</em><span class="optional">[</span>, <em>timeout</em><span class="optional">]</span><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Queue.get" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Remove and return an item from the queue. If optional args <em>block</em> is
|
||
<code class="docutils literal notranslate"><span class="pre">True</span></code> (the default) and <em>timeout</em> is <code class="docutils literal notranslate"><span class="pre">None</span></code> (the default), block if
|
||
necessary until an item is available. If <em>timeout</em> is a positive number,
|
||
it blocks at most <em>timeout</em> seconds and raises the <a class="reference internal" href="queue.html#queue.Empty" title="queue.Empty"><code class="xref py py-exc docutils literal notranslate"><span class="pre">queue.Empty</span></code></a>
|
||
exception if no item was available within that time. Otherwise (block is
|
||
<code class="docutils literal notranslate"><span class="pre">False</span></code>), return an item if one is immediately available, else raise the
|
||
<a class="reference internal" href="queue.html#queue.Empty" title="queue.Empty"><code class="xref py py-exc docutils literal notranslate"><span class="pre">queue.Empty</span></code></a> exception (<em>timeout</em> is ignored in that case).</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.Queue.get_nowait">
|
||
<code class="descname">get_nowait</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Queue.get_nowait" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Equivalent to <code class="docutils literal notranslate"><span class="pre">get(False)</span></code>.</p>
|
||
</dd></dl>
|
||
|
||
<p><a class="reference internal" href="#multiprocessing.Queue" title="multiprocessing.Queue"><code class="xref py py-class docutils literal notranslate"><span class="pre">multiprocessing.Queue</span></code></a> has a few additional methods not found in
|
||
<a class="reference internal" href="queue.html#queue.Queue" title="queue.Queue"><code class="xref py py-class docutils literal notranslate"><span class="pre">queue.Queue</span></code></a>. These methods are usually unnecessary for most
|
||
code:</p>
|
||
<dl class="method">
|
||
<dt id="multiprocessing.Queue.close">
|
||
<code class="descname">close</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Queue.close" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Indicate that no more data will be put on this queue by the current
|
||
process. The background thread will quit once it has flushed all buffered
|
||
data to the pipe. This is called automatically when the queue is garbage
|
||
collected.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.Queue.join_thread">
|
||
<code class="descname">join_thread</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Queue.join_thread" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Join the background thread. This can only be used after <a class="reference internal" href="#multiprocessing.Queue.close" title="multiprocessing.Queue.close"><code class="xref py py-meth docutils literal notranslate"><span class="pre">close()</span></code></a> has
|
||
been called. It blocks until the background thread exits, ensuring that
|
||
all data in the buffer has been flushed to the pipe.</p>
|
||
<p>By default if a process is not the creator of the queue then on exit it
|
||
will attempt to join the queue’s background thread. The process can call
|
||
<a class="reference internal" href="#multiprocessing.Queue.cancel_join_thread" title="multiprocessing.Queue.cancel_join_thread"><code class="xref py py-meth docutils literal notranslate"><span class="pre">cancel_join_thread()</span></code></a> to make <a class="reference internal" href="#multiprocessing.Queue.join_thread" title="multiprocessing.Queue.join_thread"><code class="xref py py-meth docutils literal notranslate"><span class="pre">join_thread()</span></code></a> do nothing.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.Queue.cancel_join_thread">
|
||
<code class="descname">cancel_join_thread</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Queue.cancel_join_thread" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Prevent <a class="reference internal" href="#multiprocessing.Queue.join_thread" title="multiprocessing.Queue.join_thread"><code class="xref py py-meth docutils literal notranslate"><span class="pre">join_thread()</span></code></a> from blocking. In particular, this prevents
|
||
the background thread from being joined automatically when the process
|
||
exits – see <a class="reference internal" href="#multiprocessing.Queue.join_thread" title="multiprocessing.Queue.join_thread"><code class="xref py py-meth docutils literal notranslate"><span class="pre">join_thread()</span></code></a>.</p>
|
||
<p>A better name for this method might be
|
||
<code class="docutils literal notranslate"><span class="pre">allow_exit_without_flush()</span></code>. It is likely to cause enqueued
|
||
data to lost, and you almost certainly will not need to use it.
|
||
It is really only there if you need the current process to exit
|
||
immediately without waiting to flush enqueued data to the
|
||
underlying pipe, and you don’t care about lost data.</p>
|
||
</dd></dl>
|
||
|
||
<div class="admonition note">
|
||
<p class="admonition-title">Note</p>
|
||
<p>This class’s functionality requires a functioning shared semaphore
|
||
implementation on the host operating system. Without one, the
|
||
functionality in this class will be disabled, and attempts to
|
||
instantiate a <a class="reference internal" href="#multiprocessing.Queue" title="multiprocessing.Queue"><code class="xref py py-class docutils literal notranslate"><span class="pre">Queue</span></code></a> will result in an <a class="reference internal" href="exceptions.html#ImportError" title="ImportError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">ImportError</span></code></a>. See
|
||
<a class="reference external" href="https://bugs.python.org/issue3770">bpo-3770</a> for additional information. The same holds true for any
|
||
of the specialized queue types listed below.</p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="class">
|
||
<dt id="multiprocessing.SimpleQueue">
|
||
<em class="property">class </em><code class="descclassname">multiprocessing.</code><code class="descname">SimpleQueue</code><a class="headerlink" href="#multiprocessing.SimpleQueue" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>It is a simplified <a class="reference internal" href="#multiprocessing.Queue" title="multiprocessing.Queue"><code class="xref py py-class docutils literal notranslate"><span class="pre">Queue</span></code></a> type, very close to a locked <a class="reference internal" href="#multiprocessing.Pipe" title="multiprocessing.Pipe"><code class="xref py py-class docutils literal notranslate"><span class="pre">Pipe</span></code></a>.</p>
|
||
<dl class="method">
|
||
<dt id="multiprocessing.SimpleQueue.empty">
|
||
<code class="descname">empty</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.SimpleQueue.empty" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return <code class="docutils literal notranslate"><span class="pre">True</span></code> if the queue is empty, <code class="docutils literal notranslate"><span class="pre">False</span></code> otherwise.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.SimpleQueue.get">
|
||
<code class="descname">get</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.SimpleQueue.get" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Remove and return an item from the queue.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.SimpleQueue.put">
|
||
<code class="descname">put</code><span class="sig-paren">(</span><em>item</em><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.SimpleQueue.put" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Put <em>item</em> into the queue.</p>
|
||
</dd></dl>
|
||
|
||
</dd></dl>
|
||
|
||
<dl class="class">
|
||
<dt id="multiprocessing.JoinableQueue">
|
||
<em class="property">class </em><code class="descclassname">multiprocessing.</code><code class="descname">JoinableQueue</code><span class="sig-paren">(</span><span class="optional">[</span><em>maxsize</em><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.JoinableQueue" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p><a class="reference internal" href="#multiprocessing.JoinableQueue" title="multiprocessing.JoinableQueue"><code class="xref py py-class docutils literal notranslate"><span class="pre">JoinableQueue</span></code></a>, a <a class="reference internal" href="#multiprocessing.Queue" title="multiprocessing.Queue"><code class="xref py py-class docutils literal notranslate"><span class="pre">Queue</span></code></a> subclass, is a queue which
|
||
additionally has <a class="reference internal" href="#multiprocessing.JoinableQueue.task_done" title="multiprocessing.JoinableQueue.task_done"><code class="xref py py-meth docutils literal notranslate"><span class="pre">task_done()</span></code></a> and <a class="reference internal" href="#multiprocessing.JoinableQueue.join" title="multiprocessing.JoinableQueue.join"><code class="xref py py-meth docutils literal notranslate"><span class="pre">join()</span></code></a> methods.</p>
|
||
<dl class="method">
|
||
<dt id="multiprocessing.JoinableQueue.task_done">
|
||
<code class="descname">task_done</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.JoinableQueue.task_done" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Indicate that a formerly enqueued task is complete. Used by queue
|
||
consumers. For each <a class="reference internal" href="#multiprocessing.Queue.get" title="multiprocessing.Queue.get"><code class="xref py py-meth docutils literal notranslate"><span class="pre">get()</span></code></a> used to fetch a task, a subsequent
|
||
call to <a class="reference internal" href="#multiprocessing.JoinableQueue.task_done" title="multiprocessing.JoinableQueue.task_done"><code class="xref py py-meth docutils literal notranslate"><span class="pre">task_done()</span></code></a> tells the queue that the processing on the task
|
||
is complete.</p>
|
||
<p>If a <a class="reference internal" href="queue.html#queue.Queue.join" title="queue.Queue.join"><code class="xref py py-meth docutils literal notranslate"><span class="pre">join()</span></code></a> is currently blocking, it will resume when all
|
||
items have been processed (meaning that a <a class="reference internal" href="#multiprocessing.JoinableQueue.task_done" title="multiprocessing.JoinableQueue.task_done"><code class="xref py py-meth docutils literal notranslate"><span class="pre">task_done()</span></code></a> call was
|
||
received for every item that had been <a class="reference internal" href="#multiprocessing.Queue.put" title="multiprocessing.Queue.put"><code class="xref py py-meth docutils literal notranslate"><span class="pre">put()</span></code></a> into the queue).</p>
|
||
<p>Raises a <a class="reference internal" href="exceptions.html#ValueError" title="ValueError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">ValueError</span></code></a> if called more times than there were items
|
||
placed in the queue.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.JoinableQueue.join">
|
||
<code class="descname">join</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.JoinableQueue.join" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Block until all items in the queue have been gotten and processed.</p>
|
||
<p>The count of unfinished tasks goes up whenever an item is added to the
|
||
queue. The count goes down whenever a consumer calls
|
||
<a class="reference internal" href="#multiprocessing.JoinableQueue.task_done" title="multiprocessing.JoinableQueue.task_done"><code class="xref py py-meth docutils literal notranslate"><span class="pre">task_done()</span></code></a> to indicate that the item was retrieved and all work on
|
||
it is complete. When the count of unfinished tasks drops to zero,
|
||
<a class="reference internal" href="queue.html#queue.Queue.join" title="queue.Queue.join"><code class="xref py py-meth docutils literal notranslate"><span class="pre">join()</span></code></a> unblocks.</p>
|
||
</dd></dl>
|
||
|
||
</dd></dl>
|
||
|
||
</div>
|
||
<div class="section" id="miscellaneous">
|
||
<h3>Miscellaneous<a class="headerlink" href="#miscellaneous" title="Permalink to this headline">¶</a></h3>
|
||
<dl class="function">
|
||
<dt id="multiprocessing.active_children">
|
||
<code class="descclassname">multiprocessing.</code><code class="descname">active_children</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.active_children" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return list of all live children of the current process.</p>
|
||
<p>Calling this has the side effect of “joining” any processes which have
|
||
already finished.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="function">
|
||
<dt id="multiprocessing.cpu_count">
|
||
<code class="descclassname">multiprocessing.</code><code class="descname">cpu_count</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.cpu_count" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return the number of CPUs in the system.</p>
|
||
<p>This number is not equivalent to the number of CPUs the current process can
|
||
use. The number of usable CPUs can be obtained with
|
||
<code class="docutils literal notranslate"><span class="pre">len(os.sched_getaffinity(0))</span></code></p>
|
||
<p>May raise <a class="reference internal" href="exceptions.html#NotImplementedError" title="NotImplementedError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">NotImplementedError</span></code></a>.</p>
|
||
<div class="admonition seealso">
|
||
<p class="admonition-title">See also</p>
|
||
<p><a class="reference internal" href="os.html#os.cpu_count" title="os.cpu_count"><code class="xref py py-func docutils literal notranslate"><span class="pre">os.cpu_count()</span></code></a></p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="function">
|
||
<dt id="multiprocessing.current_process">
|
||
<code class="descclassname">multiprocessing.</code><code class="descname">current_process</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.current_process" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return the <a class="reference internal" href="#multiprocessing.Process" title="multiprocessing.Process"><code class="xref py py-class docutils literal notranslate"><span class="pre">Process</span></code></a> object corresponding to the current process.</p>
|
||
<p>An analogue of <a class="reference internal" href="threading.html#threading.current_thread" title="threading.current_thread"><code class="xref py py-func docutils literal notranslate"><span class="pre">threading.current_thread()</span></code></a>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="function">
|
||
<dt id="multiprocessing.freeze_support">
|
||
<code class="descclassname">multiprocessing.</code><code class="descname">freeze_support</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.freeze_support" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Add support for when a program which uses <a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a> has been
|
||
frozen to produce a Windows executable. (Has been tested with <strong>py2exe</strong>,
|
||
<strong>PyInstaller</strong> and <strong>cx_Freeze</strong>.)</p>
|
||
<p>One needs to call this function straight after the <code class="docutils literal notranslate"><span class="pre">if</span> <span class="pre">__name__</span> <span class="pre">==</span>
|
||
<span class="pre">'__main__'</span></code> line of the main module. For example:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Process</span><span class="p">,</span> <span class="n">freeze_support</span>
|
||
|
||
<span class="k">def</span> <span class="nf">f</span><span class="p">():</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'hello world!'</span><span class="p">)</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="n">freeze_support</span><span class="p">()</span>
|
||
<span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">f</span><span class="p">)</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>If the <code class="docutils literal notranslate"><span class="pre">freeze_support()</span></code> line is omitted then trying to run the frozen
|
||
executable will raise <a class="reference internal" href="exceptions.html#RuntimeError" title="RuntimeError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">RuntimeError</span></code></a>.</p>
|
||
<p>Calling <code class="docutils literal notranslate"><span class="pre">freeze_support()</span></code> has no effect when invoked on any operating
|
||
system other than Windows. In addition, if the module is being run
|
||
normally by the Python interpreter on Windows (the program has not been
|
||
frozen), then <code class="docutils literal notranslate"><span class="pre">freeze_support()</span></code> has no effect.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="function">
|
||
<dt id="multiprocessing.get_all_start_methods">
|
||
<code class="descclassname">multiprocessing.</code><code class="descname">get_all_start_methods</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.get_all_start_methods" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Returns a list of the supported start methods, the first of which
|
||
is the default. The possible start methods are <code class="docutils literal notranslate"><span class="pre">'fork'</span></code>,
|
||
<code class="docutils literal notranslate"><span class="pre">'spawn'</span></code> and <code class="docutils literal notranslate"><span class="pre">'forkserver'</span></code>. On Windows only <code class="docutils literal notranslate"><span class="pre">'spawn'</span></code> is
|
||
available. On Unix <code class="docutils literal notranslate"><span class="pre">'fork'</span></code> and <code class="docutils literal notranslate"><span class="pre">'spawn'</span></code> are always
|
||
supported, with <code class="docutils literal notranslate"><span class="pre">'fork'</span></code> being the default.</p>
|
||
<div class="versionadded">
|
||
<p><span class="versionmodified added">New in version 3.4.</span></p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="function">
|
||
<dt id="multiprocessing.get_context">
|
||
<code class="descclassname">multiprocessing.</code><code class="descname">get_context</code><span class="sig-paren">(</span><em>method=None</em><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.get_context" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return a context object which has the same attributes as the
|
||
<a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a> module.</p>
|
||
<p>If <em>method</em> is <code class="docutils literal notranslate"><span class="pre">None</span></code> then the default context is returned.
|
||
Otherwise <em>method</em> should be <code class="docutils literal notranslate"><span class="pre">'fork'</span></code>, <code class="docutils literal notranslate"><span class="pre">'spawn'</span></code>,
|
||
<code class="docutils literal notranslate"><span class="pre">'forkserver'</span></code>. <a class="reference internal" href="exceptions.html#ValueError" title="ValueError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">ValueError</span></code></a> is raised if the specified
|
||
start method is not available.</p>
|
||
<div class="versionadded">
|
||
<p><span class="versionmodified added">New in version 3.4.</span></p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="function">
|
||
<dt id="multiprocessing.get_start_method">
|
||
<code class="descclassname">multiprocessing.</code><code class="descname">get_start_method</code><span class="sig-paren">(</span><em>allow_none=False</em><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.get_start_method" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return the name of start method used for starting processes.</p>
|
||
<p>If the start method has not been fixed and <em>allow_none</em> is false,
|
||
then the start method is fixed to the default and the name is
|
||
returned. If the start method has not been fixed and <em>allow_none</em>
|
||
is true then <code class="docutils literal notranslate"><span class="pre">None</span></code> is returned.</p>
|
||
<p>The return value can be <code class="docutils literal notranslate"><span class="pre">'fork'</span></code>, <code class="docutils literal notranslate"><span class="pre">'spawn'</span></code>, <code class="docutils literal notranslate"><span class="pre">'forkserver'</span></code>
|
||
or <code class="docutils literal notranslate"><span class="pre">None</span></code>. <code class="docutils literal notranslate"><span class="pre">'fork'</span></code> is the default on Unix, while <code class="docutils literal notranslate"><span class="pre">'spawn'</span></code> is
|
||
the default on Windows.</p>
|
||
<div class="versionadded">
|
||
<p><span class="versionmodified added">New in version 3.4.</span></p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="function">
|
||
<dt id="multiprocessing.set_executable">
|
||
<code class="descclassname">multiprocessing.</code><code class="descname">set_executable</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.set_executable" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Sets the path of the Python interpreter to use when starting a child process.
|
||
(By default <a class="reference internal" href="sys.html#sys.executable" title="sys.executable"><code class="xref py py-data docutils literal notranslate"><span class="pre">sys.executable</span></code></a> is used). Embedders will probably need to
|
||
do some thing like</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="n">set_executable</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">exec_prefix</span><span class="p">,</span> <span class="s1">'pythonw.exe'</span><span class="p">))</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>before they can create child processes.</p>
|
||
<div class="versionchanged">
|
||
<p><span class="versionmodified changed">Changed in version 3.4: </span>Now supported on Unix when the <code class="docutils literal notranslate"><span class="pre">'spawn'</span></code> start method is used.</p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="function">
|
||
<dt id="multiprocessing.set_start_method">
|
||
<code class="descclassname">multiprocessing.</code><code class="descname">set_start_method</code><span class="sig-paren">(</span><em>method</em><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.set_start_method" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Set the method which should be used to start child processes.
|
||
<em>method</em> can be <code class="docutils literal notranslate"><span class="pre">'fork'</span></code>, <code class="docutils literal notranslate"><span class="pre">'spawn'</span></code> or <code class="docutils literal notranslate"><span class="pre">'forkserver'</span></code>.</p>
|
||
<p>Note that this should be called at most once, and it should be
|
||
protected inside the <code class="docutils literal notranslate"><span class="pre">if</span> <span class="pre">__name__</span> <span class="pre">==</span> <span class="pre">'__main__'</span></code> clause of the
|
||
main module.</p>
|
||
<div class="versionadded">
|
||
<p><span class="versionmodified added">New in version 3.4.</span></p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<div class="admonition note">
|
||
<p class="admonition-title">Note</p>
|
||
<p><a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a> contains no analogues of
|
||
<a class="reference internal" href="threading.html#threading.active_count" title="threading.active_count"><code class="xref py py-func docutils literal notranslate"><span class="pre">threading.active_count()</span></code></a>, <a class="reference internal" href="threading.html#threading.enumerate" title="threading.enumerate"><code class="xref py py-func docutils literal notranslate"><span class="pre">threading.enumerate()</span></code></a>,
|
||
<a class="reference internal" href="threading.html#threading.settrace" title="threading.settrace"><code class="xref py py-func docutils literal notranslate"><span class="pre">threading.settrace()</span></code></a>, <a class="reference internal" href="threading.html#threading.setprofile" title="threading.setprofile"><code class="xref py py-func docutils literal notranslate"><span class="pre">threading.setprofile()</span></code></a>,
|
||
<a class="reference internal" href="threading.html#threading.Timer" title="threading.Timer"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.Timer</span></code></a>, or <a class="reference internal" href="threading.html#threading.local" title="threading.local"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.local</span></code></a>.</p>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="connection-objects">
|
||
<h3>Connection Objects<a class="headerlink" href="#connection-objects" title="Permalink to this headline">¶</a></h3>
|
||
<p>Connection objects allow the sending and receiving of picklable objects or
|
||
strings. They can be thought of as message oriented connected sockets.</p>
|
||
<p>Connection objects are usually created using
|
||
<a class="reference internal" href="#multiprocessing.Pipe" title="multiprocessing.Pipe"><code class="xref py py-func docutils literal notranslate"><span class="pre">Pipe</span></code></a> – see also
|
||
<a class="reference internal" href="#multiprocessing-listeners-clients"><span class="std std-ref">Listeners and Clients</span></a>.</p>
|
||
<dl class="class">
|
||
<dt id="multiprocessing.connection.Connection">
|
||
<em class="property">class </em><code class="descclassname">multiprocessing.connection.</code><code class="descname">Connection</code><a class="headerlink" href="#multiprocessing.connection.Connection" title="Permalink to this definition">¶</a></dt>
|
||
<dd><dl class="method">
|
||
<dt id="multiprocessing.connection.Connection.send">
|
||
<code class="descname">send</code><span class="sig-paren">(</span><em>obj</em><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.connection.Connection.send" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Send an object to the other end of the connection which should be read
|
||
using <a class="reference internal" href="#multiprocessing.connection.Connection.recv" title="multiprocessing.connection.Connection.recv"><code class="xref py py-meth docutils literal notranslate"><span class="pre">recv()</span></code></a>.</p>
|
||
<p>The object must be picklable. Very large pickles (approximately 32 MiB+,
|
||
though it depends on the OS) may raise a <a class="reference internal" href="exceptions.html#ValueError" title="ValueError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">ValueError</span></code></a> exception.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.connection.Connection.recv">
|
||
<code class="descname">recv</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.connection.Connection.recv" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return an object sent from the other end of the connection using
|
||
<a class="reference internal" href="#multiprocessing.connection.Connection.send" title="multiprocessing.connection.Connection.send"><code class="xref py py-meth docutils literal notranslate"><span class="pre">send()</span></code></a>. Blocks until there is something to receive. Raises
|
||
<a class="reference internal" href="exceptions.html#EOFError" title="EOFError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">EOFError</span></code></a> if there is nothing left to receive
|
||
and the other end was closed.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.connection.Connection.fileno">
|
||
<code class="descname">fileno</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.connection.Connection.fileno" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return the file descriptor or handle used by the connection.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.connection.Connection.close">
|
||
<code class="descname">close</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.connection.Connection.close" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Close the connection.</p>
|
||
<p>This is called automatically when the connection is garbage collected.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.connection.Connection.poll">
|
||
<code class="descname">poll</code><span class="sig-paren">(</span><span class="optional">[</span><em>timeout</em><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.connection.Connection.poll" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return whether there is any data available to be read.</p>
|
||
<p>If <em>timeout</em> is not specified then it will return immediately. If
|
||
<em>timeout</em> is a number then this specifies the maximum time in seconds to
|
||
block. If <em>timeout</em> is <code class="docutils literal notranslate"><span class="pre">None</span></code> then an infinite timeout is used.</p>
|
||
<p>Note that multiple connection objects may be polled at once by
|
||
using <a class="reference internal" href="#multiprocessing.connection.wait" title="multiprocessing.connection.wait"><code class="xref py py-func docutils literal notranslate"><span class="pre">multiprocessing.connection.wait()</span></code></a>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.connection.Connection.send_bytes">
|
||
<code class="descname">send_bytes</code><span class="sig-paren">(</span><em>buffer</em><span class="optional">[</span>, <em>offset</em><span class="optional">[</span>, <em>size</em><span class="optional">]</span><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.connection.Connection.send_bytes" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Send byte data from a <a class="reference internal" href="../glossary.html#term-bytes-like-object"><span class="xref std std-term">bytes-like object</span></a> as a complete message.</p>
|
||
<p>If <em>offset</em> is given then data is read from that position in <em>buffer</em>. If
|
||
<em>size</em> is given then that many bytes will be read from buffer. Very large
|
||
buffers (approximately 32 MiB+, though it depends on the OS) may raise a
|
||
<a class="reference internal" href="exceptions.html#ValueError" title="ValueError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">ValueError</span></code></a> exception</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.connection.Connection.recv_bytes">
|
||
<code class="descname">recv_bytes</code><span class="sig-paren">(</span><span class="optional">[</span><em>maxlength</em><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.connection.Connection.recv_bytes" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return a complete message of byte data sent from the other end of the
|
||
connection as a string. Blocks until there is something to receive.
|
||
Raises <a class="reference internal" href="exceptions.html#EOFError" title="EOFError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">EOFError</span></code></a> if there is nothing left
|
||
to receive and the other end has closed.</p>
|
||
<p>If <em>maxlength</em> is specified and the message is longer than <em>maxlength</em>
|
||
then <a class="reference internal" href="exceptions.html#OSError" title="OSError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">OSError</span></code></a> is raised and the connection will no longer be
|
||
readable.</p>
|
||
<div class="versionchanged">
|
||
<p><span class="versionmodified changed">Changed in version 3.3: </span>This function used to raise <a class="reference internal" href="exceptions.html#IOError" title="IOError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">IOError</span></code></a>, which is now an
|
||
alias of <a class="reference internal" href="exceptions.html#OSError" title="OSError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">OSError</span></code></a>.</p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.connection.Connection.recv_bytes_into">
|
||
<code class="descname">recv_bytes_into</code><span class="sig-paren">(</span><em>buffer</em><span class="optional">[</span>, <em>offset</em><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.connection.Connection.recv_bytes_into" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Read into <em>buffer</em> a complete message of byte data sent from the other end
|
||
of the connection and return the number of bytes in the message. Blocks
|
||
until there is something to receive. Raises
|
||
<a class="reference internal" href="exceptions.html#EOFError" title="EOFError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">EOFError</span></code></a> if there is nothing left to receive and the other end was
|
||
closed.</p>
|
||
<p><em>buffer</em> must be a writable <a class="reference internal" href="../glossary.html#term-bytes-like-object"><span class="xref std std-term">bytes-like object</span></a>. If
|
||
<em>offset</em> is given then the message will be written into the buffer from
|
||
that position. Offset must be a non-negative integer less than the
|
||
length of <em>buffer</em> (in bytes).</p>
|
||
<p>If the buffer is too short then a <code class="xref py py-exc docutils literal notranslate"><span class="pre">BufferTooShort</span></code> exception is
|
||
raised and the complete message is available as <code class="docutils literal notranslate"><span class="pre">e.args[0]</span></code> where <code class="docutils literal notranslate"><span class="pre">e</span></code>
|
||
is the exception instance.</p>
|
||
</dd></dl>
|
||
|
||
<div class="versionchanged">
|
||
<p><span class="versionmodified changed">Changed in version 3.3: </span>Connection objects themselves can now be transferred between processes
|
||
using <a class="reference internal" href="#multiprocessing.connection.Connection.send" title="multiprocessing.connection.Connection.send"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Connection.send()</span></code></a> and <a class="reference internal" href="#multiprocessing.connection.Connection.recv" title="multiprocessing.connection.Connection.recv"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Connection.recv()</span></code></a>.</p>
|
||
</div>
|
||
<div class="versionadded">
|
||
<p><span class="versionmodified added">New in version 3.3: </span>Connection objects now support the context management protocol – see
|
||
<a class="reference internal" href="stdtypes.html#typecontextmanager"><span class="std std-ref">Context Manager Types</span></a>. <a class="reference internal" href="stdtypes.html#contextmanager.__enter__" title="contextmanager.__enter__"><code class="xref py py-meth docutils literal notranslate"><span class="pre">__enter__()</span></code></a> returns the
|
||
connection object, and <a class="reference internal" href="stdtypes.html#contextmanager.__exit__" title="contextmanager.__exit__"><code class="xref py py-meth docutils literal notranslate"><span class="pre">__exit__()</span></code></a> calls <a class="reference internal" href="#multiprocessing.connection.Connection.close" title="multiprocessing.connection.Connection.close"><code class="xref py py-meth docutils literal notranslate"><span class="pre">close()</span></code></a>.</p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<p>For example:</p>
|
||
<div class="highlight-pycon3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Pipe</span>
|
||
<span class="gp">>>> </span><span class="n">a</span><span class="p">,</span> <span class="n">b</span> <span class="o">=</span> <span class="n">Pipe</span><span class="p">()</span>
|
||
<span class="gp">>>> </span><span class="n">a</span><span class="o">.</span><span class="n">send</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'hello'</span><span class="p">,</span> <span class="kc">None</span><span class="p">])</span>
|
||
<span class="gp">>>> </span><span class="n">b</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
|
||
<span class="go">[1, 'hello', None]</span>
|
||
<span class="gp">>>> </span><span class="n">b</span><span class="o">.</span><span class="n">send_bytes</span><span class="p">(</span><span class="sa">b</span><span class="s1">'thank you'</span><span class="p">)</span>
|
||
<span class="gp">>>> </span><span class="n">a</span><span class="o">.</span><span class="n">recv_bytes</span><span class="p">()</span>
|
||
<span class="go">b'thank you'</span>
|
||
<span class="gp">>>> </span><span class="kn">import</span> <span class="nn">array</span>
|
||
<span class="gp">>>> </span><span class="n">arr1</span> <span class="o">=</span> <span class="n">array</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="s1">'i'</span><span class="p">,</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">))</span>
|
||
<span class="gp">>>> </span><span class="n">arr2</span> <span class="o">=</span> <span class="n">array</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="s1">'i'</span><span class="p">,</span> <span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">*</span> <span class="mi">10</span><span class="p">)</span>
|
||
<span class="gp">>>> </span><span class="n">a</span><span class="o">.</span><span class="n">send_bytes</span><span class="p">(</span><span class="n">arr1</span><span class="p">)</span>
|
||
<span class="gp">>>> </span><span class="n">count</span> <span class="o">=</span> <span class="n">b</span><span class="o">.</span><span class="n">recv_bytes_into</span><span class="p">(</span><span class="n">arr2</span><span class="p">)</span>
|
||
<span class="gp">>>> </span><span class="k">assert</span> <span class="n">count</span> <span class="o">==</span> <span class="nb">len</span><span class="p">(</span><span class="n">arr1</span><span class="p">)</span> <span class="o">*</span> <span class="n">arr1</span><span class="o">.</span><span class="n">itemsize</span>
|
||
<span class="gp">>>> </span><span class="n">arr2</span>
|
||
<span class="go">array('i', [0, 1, 2, 3, 4, 0, 0, 0, 0, 0])</span>
|
||
</pre></div>
|
||
</div>
|
||
<div class="admonition warning">
|
||
<p class="admonition-title">Warning</p>
|
||
<p>The <a class="reference internal" href="#multiprocessing.connection.Connection.recv" title="multiprocessing.connection.Connection.recv"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Connection.recv()</span></code></a> method automatically unpickles the data it
|
||
receives, which can be a security risk unless you can trust the process
|
||
which sent the message.</p>
|
||
<p>Therefore, unless the connection object was produced using <code class="xref py py-func docutils literal notranslate"><span class="pre">Pipe()</span></code> you
|
||
should only use the <a class="reference internal" href="#multiprocessing.connection.Connection.recv" title="multiprocessing.connection.Connection.recv"><code class="xref py py-meth docutils literal notranslate"><span class="pre">recv()</span></code></a> and <a class="reference internal" href="#multiprocessing.connection.Connection.send" title="multiprocessing.connection.Connection.send"><code class="xref py py-meth docutils literal notranslate"><span class="pre">send()</span></code></a>
|
||
methods after performing some sort of authentication. See
|
||
<a class="reference internal" href="#multiprocessing-auth-keys"><span class="std std-ref">Authentication keys</span></a>.</p>
|
||
</div>
|
||
<div class="admonition warning">
|
||
<p class="admonition-title">Warning</p>
|
||
<p>If a process is killed while it is trying to read or write to a pipe then
|
||
the data in the pipe is likely to become corrupted, because it may become
|
||
impossible to be sure where the message boundaries lie.</p>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="synchronization-primitives">
|
||
<h3>Synchronization primitives<a class="headerlink" href="#synchronization-primitives" title="Permalink to this headline">¶</a></h3>
|
||
<p>Generally synchronization primitives are not as necessary in a multiprocess
|
||
program as they are in a multithreaded program. See the documentation for
|
||
<a class="reference internal" href="threading.html#module-threading" title="threading: Thread-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">threading</span></code></a> module.</p>
|
||
<p>Note that one can also create synchronization primitives by using a manager
|
||
object – see <a class="reference internal" href="#multiprocessing-managers"><span class="std std-ref">Managers</span></a>.</p>
|
||
<dl class="class">
|
||
<dt id="multiprocessing.Barrier">
|
||
<em class="property">class </em><code class="descclassname">multiprocessing.</code><code class="descname">Barrier</code><span class="sig-paren">(</span><em>parties</em><span class="optional">[</span>, <em>action</em><span class="optional">[</span>, <em>timeout</em><span class="optional">]</span><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Barrier" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>A barrier object: a clone of <a class="reference internal" href="threading.html#threading.Barrier" title="threading.Barrier"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.Barrier</span></code></a>.</p>
|
||
<div class="versionadded">
|
||
<p><span class="versionmodified added">New in version 3.3.</span></p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="class">
|
||
<dt id="multiprocessing.BoundedSemaphore">
|
||
<em class="property">class </em><code class="descclassname">multiprocessing.</code><code class="descname">BoundedSemaphore</code><span class="sig-paren">(</span><span class="optional">[</span><em>value</em><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.BoundedSemaphore" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>A bounded semaphore object: a close analog of
|
||
<a class="reference internal" href="threading.html#threading.BoundedSemaphore" title="threading.BoundedSemaphore"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.BoundedSemaphore</span></code></a>.</p>
|
||
<p>A solitary difference from its close analog exists: its <code class="docutils literal notranslate"><span class="pre">acquire</span></code> method’s
|
||
first argument is named <em>block</em>, as is consistent with <a class="reference internal" href="#multiprocessing.Lock.acquire" title="multiprocessing.Lock.acquire"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Lock.acquire()</span></code></a>.</p>
|
||
<div class="admonition note">
|
||
<p class="admonition-title">Note</p>
|
||
<p>On Mac OS X, this is indistinguishable from <a class="reference internal" href="#multiprocessing.Semaphore" title="multiprocessing.Semaphore"><code class="xref py py-class docutils literal notranslate"><span class="pre">Semaphore</span></code></a> because
|
||
<code class="docutils literal notranslate"><span class="pre">sem_getvalue()</span></code> is not implemented on that platform.</p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="class">
|
||
<dt id="multiprocessing.Condition">
|
||
<em class="property">class </em><code class="descclassname">multiprocessing.</code><code class="descname">Condition</code><span class="sig-paren">(</span><span class="optional">[</span><em>lock</em><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Condition" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>A condition variable: an alias for <a class="reference internal" href="threading.html#threading.Condition" title="threading.Condition"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.Condition</span></code></a>.</p>
|
||
<p>If <em>lock</em> is specified then it should be a <a class="reference internal" href="#multiprocessing.Lock" title="multiprocessing.Lock"><code class="xref py py-class docutils literal notranslate"><span class="pre">Lock</span></code></a> or <a class="reference internal" href="#multiprocessing.RLock" title="multiprocessing.RLock"><code class="xref py py-class docutils literal notranslate"><span class="pre">RLock</span></code></a>
|
||
object from <a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a>.</p>
|
||
<div class="versionchanged">
|
||
<p><span class="versionmodified changed">Changed in version 3.3: </span>The <a class="reference internal" href="threading.html#threading.Condition.wait_for" title="threading.Condition.wait_for"><code class="xref py py-meth docutils literal notranslate"><span class="pre">wait_for()</span></code></a> method was added.</p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="class">
|
||
<dt id="multiprocessing.Event">
|
||
<em class="property">class </em><code class="descclassname">multiprocessing.</code><code class="descname">Event</code><a class="headerlink" href="#multiprocessing.Event" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>A clone of <a class="reference internal" href="threading.html#threading.Event" title="threading.Event"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.Event</span></code></a>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="class">
|
||
<dt id="multiprocessing.Lock">
|
||
<em class="property">class </em><code class="descclassname">multiprocessing.</code><code class="descname">Lock</code><a class="headerlink" href="#multiprocessing.Lock" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>A non-recursive lock object: a close analog of <a class="reference internal" href="threading.html#threading.Lock" title="threading.Lock"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.Lock</span></code></a>.
|
||
Once a process or thread has acquired a lock, subsequent attempts to
|
||
acquire it from any process or thread will block until it is released;
|
||
any process or thread may release it. The concepts and behaviors of
|
||
<a class="reference internal" href="threading.html#threading.Lock" title="threading.Lock"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.Lock</span></code></a> as it applies to threads are replicated here in
|
||
<a class="reference internal" href="#multiprocessing.Lock" title="multiprocessing.Lock"><code class="xref py py-class docutils literal notranslate"><span class="pre">multiprocessing.Lock</span></code></a> as it applies to either processes or threads,
|
||
except as noted.</p>
|
||
<p>Note that <a class="reference internal" href="#multiprocessing.Lock" title="multiprocessing.Lock"><code class="xref py py-class docutils literal notranslate"><span class="pre">Lock</span></code></a> is actually a factory function which returns an
|
||
instance of <code class="docutils literal notranslate"><span class="pre">multiprocessing.synchronize.Lock</span></code> initialized with a
|
||
default context.</p>
|
||
<p><a class="reference internal" href="#multiprocessing.Lock" title="multiprocessing.Lock"><code class="xref py py-class docutils literal notranslate"><span class="pre">Lock</span></code></a> supports the <a class="reference internal" href="../glossary.html#term-context-manager"><span class="xref std std-term">context manager</span></a> protocol and thus may be
|
||
used in <a class="reference internal" href="../reference/compound_stmts.html#with"><code class="xref std std-keyword docutils literal notranslate"><span class="pre">with</span></code></a> statements.</p>
|
||
<dl class="method">
|
||
<dt id="multiprocessing.Lock.acquire">
|
||
<code class="descname">acquire</code><span class="sig-paren">(</span><em>block=True</em>, <em>timeout=None</em><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Lock.acquire" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Acquire a lock, blocking or non-blocking.</p>
|
||
<p>With the <em>block</em> argument set to <code class="docutils literal notranslate"><span class="pre">True</span></code> (the default), the method call
|
||
will block until the lock is in an unlocked state, then set it to locked
|
||
and return <code class="docutils literal notranslate"><span class="pre">True</span></code>. Note that the name of this first argument differs
|
||
from that in <a class="reference internal" href="threading.html#threading.Lock.acquire" title="threading.Lock.acquire"><code class="xref py py-meth docutils literal notranslate"><span class="pre">threading.Lock.acquire()</span></code></a>.</p>
|
||
<p>With the <em>block</em> argument set to <code class="docutils literal notranslate"><span class="pre">False</span></code>, the method call does not
|
||
block. If the lock is currently in a locked state, return <code class="docutils literal notranslate"><span class="pre">False</span></code>;
|
||
otherwise set the lock to a locked state and return <code class="docutils literal notranslate"><span class="pre">True</span></code>.</p>
|
||
<p>When invoked with a positive, floating-point value for <em>timeout</em>, block
|
||
for at most the number of seconds specified by <em>timeout</em> as long as
|
||
the lock can not be acquired. Invocations with a negative value for
|
||
<em>timeout</em> are equivalent to a <em>timeout</em> of zero. Invocations with a
|
||
<em>timeout</em> value of <code class="docutils literal notranslate"><span class="pre">None</span></code> (the default) set the timeout period to
|
||
infinite. Note that the treatment of negative or <code class="docutils literal notranslate"><span class="pre">None</span></code> values for
|
||
<em>timeout</em> differs from the implemented behavior in
|
||
<a class="reference internal" href="threading.html#threading.Lock.acquire" title="threading.Lock.acquire"><code class="xref py py-meth docutils literal notranslate"><span class="pre">threading.Lock.acquire()</span></code></a>. The <em>timeout</em> argument has no practical
|
||
implications if the <em>block</em> argument is set to <code class="docutils literal notranslate"><span class="pre">False</span></code> and is thus
|
||
ignored. Returns <code class="docutils literal notranslate"><span class="pre">True</span></code> if the lock has been acquired or <code class="docutils literal notranslate"><span class="pre">False</span></code> if
|
||
the timeout period has elapsed.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.Lock.release">
|
||
<code class="descname">release</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Lock.release" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Release a lock. This can be called from any process or thread, not only
|
||
the process or thread which originally acquired the lock.</p>
|
||
<p>Behavior is the same as in <a class="reference internal" href="threading.html#threading.Lock.release" title="threading.Lock.release"><code class="xref py py-meth docutils literal notranslate"><span class="pre">threading.Lock.release()</span></code></a> except that
|
||
when invoked on an unlocked lock, a <a class="reference internal" href="exceptions.html#ValueError" title="ValueError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">ValueError</span></code></a> is raised.</p>
|
||
</dd></dl>
|
||
|
||
</dd></dl>
|
||
|
||
<dl class="class">
|
||
<dt id="multiprocessing.RLock">
|
||
<em class="property">class </em><code class="descclassname">multiprocessing.</code><code class="descname">RLock</code><a class="headerlink" href="#multiprocessing.RLock" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>A recursive lock object: a close analog of <a class="reference internal" href="threading.html#threading.RLock" title="threading.RLock"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.RLock</span></code></a>. A
|
||
recursive lock must be released by the process or thread that acquired it.
|
||
Once a process or thread has acquired a recursive lock, the same process
|
||
or thread may acquire it again without blocking; that process or thread
|
||
must release it once for each time it has been acquired.</p>
|
||
<p>Note that <a class="reference internal" href="#multiprocessing.RLock" title="multiprocessing.RLock"><code class="xref py py-class docutils literal notranslate"><span class="pre">RLock</span></code></a> is actually a factory function which returns an
|
||
instance of <code class="docutils literal notranslate"><span class="pre">multiprocessing.synchronize.RLock</span></code> initialized with a
|
||
default context.</p>
|
||
<p><a class="reference internal" href="#multiprocessing.RLock" title="multiprocessing.RLock"><code class="xref py py-class docutils literal notranslate"><span class="pre">RLock</span></code></a> supports the <a class="reference internal" href="../glossary.html#term-context-manager"><span class="xref std std-term">context manager</span></a> protocol and thus may be
|
||
used in <a class="reference internal" href="../reference/compound_stmts.html#with"><code class="xref std std-keyword docutils literal notranslate"><span class="pre">with</span></code></a> statements.</p>
|
||
<dl class="method">
|
||
<dt id="multiprocessing.RLock.acquire">
|
||
<code class="descname">acquire</code><span class="sig-paren">(</span><em>block=True</em>, <em>timeout=None</em><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.RLock.acquire" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Acquire a lock, blocking or non-blocking.</p>
|
||
<p>When invoked with the <em>block</em> argument set to <code class="docutils literal notranslate"><span class="pre">True</span></code>, block until the
|
||
lock is in an unlocked state (not owned by any process or thread) unless
|
||
the lock is already owned by the current process or thread. The current
|
||
process or thread then takes ownership of the lock (if it does not
|
||
already have ownership) and the recursion level inside the lock increments
|
||
by one, resulting in a return value of <code class="docutils literal notranslate"><span class="pre">True</span></code>. Note that there are
|
||
several differences in this first argument’s behavior compared to the
|
||
implementation of <a class="reference internal" href="threading.html#threading.RLock.acquire" title="threading.RLock.acquire"><code class="xref py py-meth docutils literal notranslate"><span class="pre">threading.RLock.acquire()</span></code></a>, starting with the name
|
||
of the argument itself.</p>
|
||
<p>When invoked with the <em>block</em> argument set to <code class="docutils literal notranslate"><span class="pre">False</span></code>, do not block.
|
||
If the lock has already been acquired (and thus is owned) by another
|
||
process or thread, the current process or thread does not take ownership
|
||
and the recursion level within the lock is not changed, resulting in
|
||
a return value of <code class="docutils literal notranslate"><span class="pre">False</span></code>. If the lock is in an unlocked state, the
|
||
current process or thread takes ownership and the recursion level is
|
||
incremented, resulting in a return value of <code class="docutils literal notranslate"><span class="pre">True</span></code>.</p>
|
||
<p>Use and behaviors of the <em>timeout</em> argument are the same as in
|
||
<a class="reference internal" href="#multiprocessing.Lock.acquire" title="multiprocessing.Lock.acquire"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Lock.acquire()</span></code></a>. Note that some of these behaviors of <em>timeout</em>
|
||
differ from the implemented behaviors in <a class="reference internal" href="threading.html#threading.RLock.acquire" title="threading.RLock.acquire"><code class="xref py py-meth docutils literal notranslate"><span class="pre">threading.RLock.acquire()</span></code></a>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.RLock.release">
|
||
<code class="descname">release</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.RLock.release" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Release a lock, decrementing the recursion level. If after the
|
||
decrement the recursion level is zero, reset the lock to unlocked (not
|
||
owned by any process or thread) and if any other processes or threads
|
||
are blocked waiting for the lock to become unlocked, allow exactly one
|
||
of them to proceed. If after the decrement the recursion level is still
|
||
nonzero, the lock remains locked and owned by the calling process or
|
||
thread.</p>
|
||
<p>Only call this method when the calling process or thread owns the lock.
|
||
An <a class="reference internal" href="exceptions.html#AssertionError" title="AssertionError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">AssertionError</span></code></a> is raised if this method is called by a process
|
||
or thread other than the owner or if the lock is in an unlocked (unowned)
|
||
state. Note that the type of exception raised in this situation
|
||
differs from the implemented behavior in <a class="reference internal" href="threading.html#threading.RLock.release" title="threading.RLock.release"><code class="xref py py-meth docutils literal notranslate"><span class="pre">threading.RLock.release()</span></code></a>.</p>
|
||
</dd></dl>
|
||
|
||
</dd></dl>
|
||
|
||
<dl class="class">
|
||
<dt id="multiprocessing.Semaphore">
|
||
<em class="property">class </em><code class="descclassname">multiprocessing.</code><code class="descname">Semaphore</code><span class="sig-paren">(</span><span class="optional">[</span><em>value</em><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Semaphore" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>A semaphore object: a close analog of <a class="reference internal" href="threading.html#threading.Semaphore" title="threading.Semaphore"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.Semaphore</span></code></a>.</p>
|
||
<p>A solitary difference from its close analog exists: its <code class="docutils literal notranslate"><span class="pre">acquire</span></code> method’s
|
||
first argument is named <em>block</em>, as is consistent with <a class="reference internal" href="#multiprocessing.Lock.acquire" title="multiprocessing.Lock.acquire"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Lock.acquire()</span></code></a>.</p>
|
||
</dd></dl>
|
||
|
||
<div class="admonition note">
|
||
<p class="admonition-title">Note</p>
|
||
<p>On Mac OS X, <code class="docutils literal notranslate"><span class="pre">sem_timedwait</span></code> is unsupported, so calling <code class="docutils literal notranslate"><span class="pre">acquire()</span></code> with
|
||
a timeout will emulate that function’s behavior using a sleeping loop.</p>
|
||
</div>
|
||
<div class="admonition note">
|
||
<p class="admonition-title">Note</p>
|
||
<p>If the SIGINT signal generated by <kbd class="kbd docutils literal notranslate">Ctrl-C</kbd> arrives while the main thread is
|
||
blocked by a call to <code class="xref py py-meth docutils literal notranslate"><span class="pre">BoundedSemaphore.acquire()</span></code>, <a class="reference internal" href="#multiprocessing.Lock.acquire" title="multiprocessing.Lock.acquire"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Lock.acquire()</span></code></a>,
|
||
<a class="reference internal" href="#multiprocessing.RLock.acquire" title="multiprocessing.RLock.acquire"><code class="xref py py-meth docutils literal notranslate"><span class="pre">RLock.acquire()</span></code></a>, <code class="xref py py-meth docutils literal notranslate"><span class="pre">Semaphore.acquire()</span></code>, <code class="xref py py-meth docutils literal notranslate"><span class="pre">Condition.acquire()</span></code>
|
||
or <code class="xref py py-meth docutils literal notranslate"><span class="pre">Condition.wait()</span></code> then the call will be immediately interrupted and
|
||
<a class="reference internal" href="exceptions.html#KeyboardInterrupt" title="KeyboardInterrupt"><code class="xref py py-exc docutils literal notranslate"><span class="pre">KeyboardInterrupt</span></code></a> will be raised.</p>
|
||
<p>This differs from the behaviour of <a class="reference internal" href="threading.html#module-threading" title="threading: Thread-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">threading</span></code></a> where SIGINT will be
|
||
ignored while the equivalent blocking calls are in progress.</p>
|
||
</div>
|
||
<div class="admonition note">
|
||
<p class="admonition-title">Note</p>
|
||
<p>Some of this package’s functionality requires a functioning shared semaphore
|
||
implementation on the host operating system. Without one, the
|
||
<code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing.synchronize</span></code> module will be disabled, and attempts to
|
||
import it will result in an <a class="reference internal" href="exceptions.html#ImportError" title="ImportError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">ImportError</span></code></a>. See
|
||
<a class="reference external" href="https://bugs.python.org/issue3770">bpo-3770</a> for additional information.</p>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="shared-ctypes-objects">
|
||
<h3>Shared <a class="reference internal" href="ctypes.html#module-ctypes" title="ctypes: A foreign function library for Python."><code class="xref py py-mod docutils literal notranslate"><span class="pre">ctypes</span></code></a> Objects<a class="headerlink" href="#shared-ctypes-objects" title="Permalink to this headline">¶</a></h3>
|
||
<p>It is possible to create shared objects using shared memory which can be
|
||
inherited by child processes.</p>
|
||
<dl class="function">
|
||
<dt id="multiprocessing.Value">
|
||
<code class="descclassname">multiprocessing.</code><code class="descname">Value</code><span class="sig-paren">(</span><em>typecode_or_type</em>, <em>*args</em>, <em>lock=True</em><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Value" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return a <a class="reference internal" href="ctypes.html#module-ctypes" title="ctypes: A foreign function library for Python."><code class="xref py py-mod docutils literal notranslate"><span class="pre">ctypes</span></code></a> object allocated from shared memory. By default the
|
||
return value is actually a synchronized wrapper for the object. The object
|
||
itself can be accessed via the <em>value</em> attribute of a <a class="reference internal" href="#multiprocessing.Value" title="multiprocessing.Value"><code class="xref py py-class docutils literal notranslate"><span class="pre">Value</span></code></a>.</p>
|
||
<p><em>typecode_or_type</em> determines the type of the returned object: it is either a
|
||
ctypes type or a one character typecode of the kind used by the <a class="reference internal" href="array.html#module-array" title="array: Space efficient arrays of uniformly typed numeric values."><code class="xref py py-mod docutils literal notranslate"><span class="pre">array</span></code></a>
|
||
module. <em>*args</em> is passed on to the constructor for the type.</p>
|
||
<p>If <em>lock</em> is <code class="docutils literal notranslate"><span class="pre">True</span></code> (the default) then a new recursive lock
|
||
object is created to synchronize access to the value. If <em>lock</em> is
|
||
a <a class="reference internal" href="#multiprocessing.Lock" title="multiprocessing.Lock"><code class="xref py py-class docutils literal notranslate"><span class="pre">Lock</span></code></a> or <a class="reference internal" href="#multiprocessing.RLock" title="multiprocessing.RLock"><code class="xref py py-class docutils literal notranslate"><span class="pre">RLock</span></code></a> object then that will be used to
|
||
synchronize access to the value. If <em>lock</em> is <code class="docutils literal notranslate"><span class="pre">False</span></code> then
|
||
access to the returned object will not be automatically protected
|
||
by a lock, so it will not necessarily be “process-safe”.</p>
|
||
<p>Operations like <code class="docutils literal notranslate"><span class="pre">+=</span></code> which involve a read and write are not
|
||
atomic. So if, for instance, you want to atomically increment a
|
||
shared value it is insufficient to just do</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="n">counter</span><span class="o">.</span><span class="n">value</span> <span class="o">+=</span> <span class="mi">1</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>Assuming the associated lock is recursive (which it is by default)
|
||
you can instead do</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="n">counter</span><span class="o">.</span><span class="n">get_lock</span><span class="p">():</span>
|
||
<span class="n">counter</span><span class="o">.</span><span class="n">value</span> <span class="o">+=</span> <span class="mi">1</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>Note that <em>lock</em> is a keyword-only argument.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="function">
|
||
<dt id="multiprocessing.Array">
|
||
<code class="descclassname">multiprocessing.</code><code class="descname">Array</code><span class="sig-paren">(</span><em>typecode_or_type</em>, <em>size_or_initializer</em>, <em>*</em>, <em>lock=True</em><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.Array" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return a ctypes array allocated from shared memory. By default the return
|
||
value is actually a synchronized wrapper for the array.</p>
|
||
<p><em>typecode_or_type</em> determines the type of the elements of the returned array:
|
||
it is either a ctypes type or a one character typecode of the kind used by
|
||
the <a class="reference internal" href="array.html#module-array" title="array: Space efficient arrays of uniformly typed numeric values."><code class="xref py py-mod docutils literal notranslate"><span class="pre">array</span></code></a> module. If <em>size_or_initializer</em> is an integer, then it
|
||
determines the length of the array, and the array will be initially zeroed.
|
||
Otherwise, <em>size_or_initializer</em> is a sequence which is used to initialize
|
||
the array and whose length determines the length of the array.</p>
|
||
<p>If <em>lock</em> is <code class="docutils literal notranslate"><span class="pre">True</span></code> (the default) then a new lock object is created to
|
||
synchronize access to the value. If <em>lock</em> is a <a class="reference internal" href="#multiprocessing.Lock" title="multiprocessing.Lock"><code class="xref py py-class docutils literal notranslate"><span class="pre">Lock</span></code></a> or
|
||
<a class="reference internal" href="#multiprocessing.RLock" title="multiprocessing.RLock"><code class="xref py py-class docutils literal notranslate"><span class="pre">RLock</span></code></a> object then that will be used to synchronize access to the
|
||
value. If <em>lock</em> is <code class="docutils literal notranslate"><span class="pre">False</span></code> then access to the returned object will not be
|
||
automatically protected by a lock, so it will not necessarily be
|
||
“process-safe”.</p>
|
||
<p>Note that <em>lock</em> is a keyword only argument.</p>
|
||
<p>Note that an array of <a class="reference internal" href="ctypes.html#ctypes.c_char" title="ctypes.c_char"><code class="xref py py-data docutils literal notranslate"><span class="pre">ctypes.c_char</span></code></a> has <em>value</em> and <em>raw</em>
|
||
attributes which allow one to use it to store and retrieve strings.</p>
|
||
</dd></dl>
|
||
|
||
<div class="section" id="module-multiprocessing.sharedctypes">
|
||
<span id="the-multiprocessing-sharedctypes-module"></span><h4>The <a class="reference internal" href="#module-multiprocessing.sharedctypes" title="multiprocessing.sharedctypes: Allocate ctypes objects from shared memory."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing.sharedctypes</span></code></a> module<a class="headerlink" href="#module-multiprocessing.sharedctypes" title="Permalink to this headline">¶</a></h4>
|
||
<p>The <a class="reference internal" href="#module-multiprocessing.sharedctypes" title="multiprocessing.sharedctypes: Allocate ctypes objects from shared memory."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing.sharedctypes</span></code></a> module provides functions for allocating
|
||
<a class="reference internal" href="ctypes.html#module-ctypes" title="ctypes: A foreign function library for Python."><code class="xref py py-mod docutils literal notranslate"><span class="pre">ctypes</span></code></a> objects from shared memory which can be inherited by child
|
||
processes.</p>
|
||
<div class="admonition note">
|
||
<p class="admonition-title">Note</p>
|
||
<p>Although it is possible to store a pointer in shared memory remember that
|
||
this will refer to a location in the address space of a specific process.
|
||
However, the pointer is quite likely to be invalid in the context of a second
|
||
process and trying to dereference the pointer from the second process may
|
||
cause a crash.</p>
|
||
</div>
|
||
<dl class="function">
|
||
<dt id="multiprocessing.sharedctypes.RawArray">
|
||
<code class="descclassname">multiprocessing.sharedctypes.</code><code class="descname">RawArray</code><span class="sig-paren">(</span><em>typecode_or_type</em>, <em>size_or_initializer</em><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.sharedctypes.RawArray" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return a ctypes array allocated from shared memory.</p>
|
||
<p><em>typecode_or_type</em> determines the type of the elements of the returned array:
|
||
it is either a ctypes type or a one character typecode of the kind used by
|
||
the <a class="reference internal" href="array.html#module-array" title="array: Space efficient arrays of uniformly typed numeric values."><code class="xref py py-mod docutils literal notranslate"><span class="pre">array</span></code></a> module. If <em>size_or_initializer</em> is an integer then it
|
||
determines the length of the array, and the array will be initially zeroed.
|
||
Otherwise <em>size_or_initializer</em> is a sequence which is used to initialize the
|
||
array and whose length determines the length of the array.</p>
|
||
<p>Note that setting and getting an element is potentially non-atomic – use
|
||
<a class="reference internal" href="#multiprocessing.sharedctypes.Array" title="multiprocessing.sharedctypes.Array"><code class="xref py py-func docutils literal notranslate"><span class="pre">Array()</span></code></a> instead to make sure that access is automatically synchronized
|
||
using a lock.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="function">
|
||
<dt id="multiprocessing.sharedctypes.RawValue">
|
||
<code class="descclassname">multiprocessing.sharedctypes.</code><code class="descname">RawValue</code><span class="sig-paren">(</span><em>typecode_or_type</em>, <em>*args</em><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.sharedctypes.RawValue" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return a ctypes object allocated from shared memory.</p>
|
||
<p><em>typecode_or_type</em> determines the type of the returned object: it is either a
|
||
ctypes type or a one character typecode of the kind used by the <a class="reference internal" href="array.html#module-array" title="array: Space efficient arrays of uniformly typed numeric values."><code class="xref py py-mod docutils literal notranslate"><span class="pre">array</span></code></a>
|
||
module. <em>*args</em> is passed on to the constructor for the type.</p>
|
||
<p>Note that setting and getting the value is potentially non-atomic – use
|
||
<a class="reference internal" href="#multiprocessing.sharedctypes.Value" title="multiprocessing.sharedctypes.Value"><code class="xref py py-func docutils literal notranslate"><span class="pre">Value()</span></code></a> instead to make sure that access is automatically synchronized
|
||
using a lock.</p>
|
||
<p>Note that an array of <a class="reference internal" href="ctypes.html#ctypes.c_char" title="ctypes.c_char"><code class="xref py py-data docutils literal notranslate"><span class="pre">ctypes.c_char</span></code></a> has <code class="docutils literal notranslate"><span class="pre">value</span></code> and <code class="docutils literal notranslate"><span class="pre">raw</span></code>
|
||
attributes which allow one to use it to store and retrieve strings – see
|
||
documentation for <a class="reference internal" href="ctypes.html#module-ctypes" title="ctypes: A foreign function library for Python."><code class="xref py py-mod docutils literal notranslate"><span class="pre">ctypes</span></code></a>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="function">
|
||
<dt id="multiprocessing.sharedctypes.Array">
|
||
<code class="descclassname">multiprocessing.sharedctypes.</code><code class="descname">Array</code><span class="sig-paren">(</span><em>typecode_or_type</em>, <em>size_or_initializer</em>, <em>*</em>, <em>lock=True</em><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.sharedctypes.Array" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>The same as <a class="reference internal" href="#multiprocessing.sharedctypes.RawArray" title="multiprocessing.sharedctypes.RawArray"><code class="xref py py-func docutils literal notranslate"><span class="pre">RawArray()</span></code></a> except that depending on the value of <em>lock</em> a
|
||
process-safe synchronization wrapper may be returned instead of a raw ctypes
|
||
array.</p>
|
||
<p>If <em>lock</em> is <code class="docutils literal notranslate"><span class="pre">True</span></code> (the default) then a new lock object is created to
|
||
synchronize access to the value. If <em>lock</em> is a
|
||
<a class="reference internal" href="#multiprocessing.Lock" title="multiprocessing.Lock"><code class="xref py py-class docutils literal notranslate"><span class="pre">Lock</span></code></a> or <a class="reference internal" href="#multiprocessing.RLock" title="multiprocessing.RLock"><code class="xref py py-class docutils literal notranslate"><span class="pre">RLock</span></code></a> object
|
||
then that will be used to synchronize access to the
|
||
value. If <em>lock</em> is <code class="docutils literal notranslate"><span class="pre">False</span></code> then access to the returned object will not be
|
||
automatically protected by a lock, so it will not necessarily be
|
||
“process-safe”.</p>
|
||
<p>Note that <em>lock</em> is a keyword-only argument.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="function">
|
||
<dt id="multiprocessing.sharedctypes.Value">
|
||
<code class="descclassname">multiprocessing.sharedctypes.</code><code class="descname">Value</code><span class="sig-paren">(</span><em>typecode_or_type</em>, <em>*args</em>, <em>lock=True</em><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.sharedctypes.Value" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>The same as <a class="reference internal" href="#multiprocessing.sharedctypes.RawValue" title="multiprocessing.sharedctypes.RawValue"><code class="xref py py-func docutils literal notranslate"><span class="pre">RawValue()</span></code></a> except that depending on the value of <em>lock</em> a
|
||
process-safe synchronization wrapper may be returned instead of a raw ctypes
|
||
object.</p>
|
||
<p>If <em>lock</em> is <code class="docutils literal notranslate"><span class="pre">True</span></code> (the default) then a new lock object is created to
|
||
synchronize access to the value. If <em>lock</em> is a <a class="reference internal" href="#multiprocessing.Lock" title="multiprocessing.Lock"><code class="xref py py-class docutils literal notranslate"><span class="pre">Lock</span></code></a> or
|
||
<a class="reference internal" href="#multiprocessing.RLock" title="multiprocessing.RLock"><code class="xref py py-class docutils literal notranslate"><span class="pre">RLock</span></code></a> object then that will be used to synchronize access to the
|
||
value. If <em>lock</em> is <code class="docutils literal notranslate"><span class="pre">False</span></code> then access to the returned object will not be
|
||
automatically protected by a lock, so it will not necessarily be
|
||
“process-safe”.</p>
|
||
<p>Note that <em>lock</em> is a keyword-only argument.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="function">
|
||
<dt id="multiprocessing.sharedctypes.copy">
|
||
<code class="descclassname">multiprocessing.sharedctypes.</code><code class="descname">copy</code><span class="sig-paren">(</span><em>obj</em><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.sharedctypes.copy" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return a ctypes object allocated from shared memory which is a copy of the
|
||
ctypes object <em>obj</em>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="function">
|
||
<dt id="multiprocessing.sharedctypes.synchronized">
|
||
<code class="descclassname">multiprocessing.sharedctypes.</code><code class="descname">synchronized</code><span class="sig-paren">(</span><em>obj</em><span class="optional">[</span>, <em>lock</em><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.sharedctypes.synchronized" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return a process-safe wrapper object for a ctypes object which uses <em>lock</em> to
|
||
synchronize access. If <em>lock</em> is <code class="docutils literal notranslate"><span class="pre">None</span></code> (the default) then a
|
||
<a class="reference internal" href="#multiprocessing.RLock" title="multiprocessing.RLock"><code class="xref py py-class docutils literal notranslate"><span class="pre">multiprocessing.RLock</span></code></a> object is created automatically.</p>
|
||
<p>A synchronized wrapper will have two methods in addition to those of the
|
||
object it wraps: <code class="xref py py-meth docutils literal notranslate"><span class="pre">get_obj()</span></code> returns the wrapped object and
|
||
<code class="xref py py-meth docutils literal notranslate"><span class="pre">get_lock()</span></code> returns the lock object used for synchronization.</p>
|
||
<p>Note that accessing the ctypes object through the wrapper can be a lot slower
|
||
than accessing the raw ctypes object.</p>
|
||
<div class="versionchanged">
|
||
<p><span class="versionmodified changed">Changed in version 3.5: </span>Synchronized objects support the <a class="reference internal" href="../glossary.html#term-context-manager"><span class="xref std std-term">context manager</span></a> protocol.</p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<p>The table below compares the syntax for creating shared ctypes objects from
|
||
shared memory with the normal ctypes syntax. (In the table <code class="docutils literal notranslate"><span class="pre">MyStruct</span></code> is some
|
||
subclass of <a class="reference internal" href="ctypes.html#ctypes.Structure" title="ctypes.Structure"><code class="xref py py-class docutils literal notranslate"><span class="pre">ctypes.Structure</span></code></a>.)</p>
|
||
<table class="docutils align-center">
|
||
<colgroup>
|
||
<col style="width: 27%" />
|
||
<col style="width: 36%" />
|
||
<col style="width: 37%" />
|
||
</colgroup>
|
||
<thead>
|
||
<tr class="row-odd"><th class="head"><p>ctypes</p></th>
|
||
<th class="head"><p>sharedctypes using type</p></th>
|
||
<th class="head"><p>sharedctypes using typecode</p></th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr class="row-even"><td><p>c_double(2.4)</p></td>
|
||
<td><p>RawValue(c_double, 2.4)</p></td>
|
||
<td><p>RawValue(‘d’, 2.4)</p></td>
|
||
</tr>
|
||
<tr class="row-odd"><td><p>MyStruct(4, 6)</p></td>
|
||
<td><p>RawValue(MyStruct, 4, 6)</p></td>
|
||
<td></td>
|
||
</tr>
|
||
<tr class="row-even"><td><p>(c_short * 7)()</p></td>
|
||
<td><p>RawArray(c_short, 7)</p></td>
|
||
<td><p>RawArray(‘h’, 7)</p></td>
|
||
</tr>
|
||
<tr class="row-odd"><td><p>(c_int * 3)(9, 2, 8)</p></td>
|
||
<td><p>RawArray(c_int, (9, 2, 8))</p></td>
|
||
<td><p>RawArray(‘i’, (9, 2, 8))</p></td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
<p>Below is an example where a number of ctypes objects are modified by a child
|
||
process:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Process</span><span class="p">,</span> <span class="n">Lock</span>
|
||
<span class="kn">from</span> <span class="nn">multiprocessing.sharedctypes</span> <span class="k">import</span> <span class="n">Value</span><span class="p">,</span> <span class="n">Array</span>
|
||
<span class="kn">from</span> <span class="nn">ctypes</span> <span class="k">import</span> <span class="n">Structure</span><span class="p">,</span> <span class="n">c_double</span>
|
||
|
||
<span class="k">class</span> <span class="nc">Point</span><span class="p">(</span><span class="n">Structure</span><span class="p">):</span>
|
||
<span class="n">_fields_</span> <span class="o">=</span> <span class="p">[(</span><span class="s1">'x'</span><span class="p">,</span> <span class="n">c_double</span><span class="p">),</span> <span class="p">(</span><span class="s1">'y'</span><span class="p">,</span> <span class="n">c_double</span><span class="p">)]</span>
|
||
|
||
<span class="k">def</span> <span class="nf">modify</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">s</span><span class="p">,</span> <span class="n">A</span><span class="p">):</span>
|
||
<span class="n">n</span><span class="o">.</span><span class="n">value</span> <span class="o">**=</span> <span class="mi">2</span>
|
||
<span class="n">x</span><span class="o">.</span><span class="n">value</span> <span class="o">**=</span> <span class="mi">2</span>
|
||
<span class="n">s</span><span class="o">.</span><span class="n">value</span> <span class="o">=</span> <span class="n">s</span><span class="o">.</span><span class="n">value</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span>
|
||
<span class="k">for</span> <span class="n">a</span> <span class="ow">in</span> <span class="n">A</span><span class="p">:</span>
|
||
<span class="n">a</span><span class="o">.</span><span class="n">x</span> <span class="o">**=</span> <span class="mi">2</span>
|
||
<span class="n">a</span><span class="o">.</span><span class="n">y</span> <span class="o">**=</span> <span class="mi">2</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="n">lock</span> <span class="o">=</span> <span class="n">Lock</span><span class="p">()</span>
|
||
|
||
<span class="n">n</span> <span class="o">=</span> <span class="n">Value</span><span class="p">(</span><span class="s1">'i'</span><span class="p">,</span> <span class="mi">7</span><span class="p">)</span>
|
||
<span class="n">x</span> <span class="o">=</span> <span class="n">Value</span><span class="p">(</span><span class="n">c_double</span><span class="p">,</span> <span class="mf">1.0</span><span class="o">/</span><span class="mf">3.0</span><span class="p">,</span> <span class="n">lock</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
|
||
<span class="n">s</span> <span class="o">=</span> <span class="n">Array</span><span class="p">(</span><span class="s1">'c'</span><span class="p">,</span> <span class="sa">b</span><span class="s1">'hello world'</span><span class="p">,</span> <span class="n">lock</span><span class="o">=</span><span class="n">lock</span><span class="p">)</span>
|
||
<span class="n">A</span> <span class="o">=</span> <span class="n">Array</span><span class="p">(</span><span class="n">Point</span><span class="p">,</span> <span class="p">[(</span><span class="mf">1.875</span><span class="p">,</span><span class="o">-</span><span class="mf">6.25</span><span class="p">),</span> <span class="p">(</span><span class="o">-</span><span class="mf">5.75</span><span class="p">,</span><span class="mf">2.0</span><span class="p">),</span> <span class="p">(</span><span class="mf">2.375</span><span class="p">,</span><span class="mf">9.5</span><span class="p">)],</span> <span class="n">lock</span><span class="o">=</span><span class="n">lock</span><span class="p">)</span>
|
||
|
||
<span class="n">p</span> <span class="o">=</span> <span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">modify</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">s</span><span class="p">,</span> <span class="n">A</span><span class="p">))</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
|
||
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">n</span><span class="o">.</span><span class="n">value</span><span class="p">)</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">x</span><span class="o">.</span><span class="n">value</span><span class="p">)</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">s</span><span class="o">.</span><span class="n">value</span><span class="p">)</span>
|
||
<span class="nb">print</span><span class="p">([(</span><span class="n">a</span><span class="o">.</span><span class="n">x</span><span class="p">,</span> <span class="n">a</span><span class="o">.</span><span class="n">y</span><span class="p">)</span> <span class="k">for</span> <span class="n">a</span> <span class="ow">in</span> <span class="n">A</span><span class="p">])</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>The results printed are</p>
|
||
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>49
|
||
0.1111111111111111
|
||
HELLO WORLD
|
||
[(3.515625, 39.0625), (33.0625, 4.0), (5.640625, 90.25)]
|
||
</pre></div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="managers">
|
||
<span id="multiprocessing-managers"></span><h3>Managers<a class="headerlink" href="#managers" title="Permalink to this headline">¶</a></h3>
|
||
<p>Managers provide a way to create data which can be shared between different
|
||
processes, including sharing over a network between processes running on
|
||
different machines. A manager object controls a server process which manages
|
||
<em>shared objects</em>. Other processes can access the shared objects by using
|
||
proxies.</p>
|
||
<dl class="function">
|
||
<dt id="multiprocessing.sharedctypes.multiprocessing.Manager">
|
||
<code class="descclassname">multiprocessing.</code><code class="descname">Manager</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.sharedctypes.multiprocessing.Manager" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Returns a started <a class="reference internal" href="#multiprocessing.managers.SyncManager" title="multiprocessing.managers.SyncManager"><code class="xref py py-class docutils literal notranslate"><span class="pre">SyncManager</span></code></a> object which
|
||
can be used for sharing objects between processes. The returned manager
|
||
object corresponds to a spawned child process and has methods which will
|
||
create shared objects and return corresponding proxies.</p>
|
||
</dd></dl>
|
||
|
||
<span class="target" id="module-multiprocessing.managers"></span><p>Manager processes will be shutdown as soon as they are garbage collected or
|
||
their parent process exits. The manager classes are defined in the
|
||
<a class="reference internal" href="#module-multiprocessing.managers" title="multiprocessing.managers: Share data between process with shared objects."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing.managers</span></code></a> module:</p>
|
||
<dl class="class">
|
||
<dt id="multiprocessing.managers.BaseManager">
|
||
<em class="property">class </em><code class="descclassname">multiprocessing.managers.</code><code class="descname">BaseManager</code><span class="sig-paren">(</span><span class="optional">[</span><em>address</em><span class="optional">[</span>, <em>authkey</em><span class="optional">]</span><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.BaseManager" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Create a BaseManager object.</p>
|
||
<p>Once created one should call <a class="reference internal" href="#multiprocessing.managers.BaseManager.start" title="multiprocessing.managers.BaseManager.start"><code class="xref py py-meth docutils literal notranslate"><span class="pre">start()</span></code></a> or <code class="docutils literal notranslate"><span class="pre">get_server().serve_forever()</span></code> to ensure
|
||
that the manager object refers to a started manager process.</p>
|
||
<p><em>address</em> is the address on which the manager process listens for new
|
||
connections. If <em>address</em> is <code class="docutils literal notranslate"><span class="pre">None</span></code> then an arbitrary one is chosen.</p>
|
||
<p><em>authkey</em> is the authentication key which will be used to check the
|
||
validity of incoming connections to the server process. If
|
||
<em>authkey</em> is <code class="docutils literal notranslate"><span class="pre">None</span></code> then <code class="docutils literal notranslate"><span class="pre">current_process().authkey</span></code> is used.
|
||
Otherwise <em>authkey</em> is used and it must be a byte string.</p>
|
||
<dl class="method">
|
||
<dt id="multiprocessing.managers.BaseManager.start">
|
||
<code class="descname">start</code><span class="sig-paren">(</span><span class="optional">[</span><em>initializer</em><span class="optional">[</span>, <em>initargs</em><span class="optional">]</span><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.BaseManager.start" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Start a subprocess to start the manager. If <em>initializer</em> is not <code class="docutils literal notranslate"><span class="pre">None</span></code>
|
||
then the subprocess will call <code class="docutils literal notranslate"><span class="pre">initializer(*initargs)</span></code> when it starts.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.managers.BaseManager.get_server">
|
||
<code class="descname">get_server</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.BaseManager.get_server" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Returns a <code class="xref py py-class docutils literal notranslate"><span class="pre">Server</span></code> object which represents the actual server under
|
||
the control of the Manager. The <code class="xref py py-class docutils literal notranslate"><span class="pre">Server</span></code> object supports the
|
||
<code class="xref py py-meth docutils literal notranslate"><span class="pre">serve_forever()</span></code> method:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">multiprocessing.managers</span> <span class="k">import</span> <span class="n">BaseManager</span>
|
||
<span class="gp">>>> </span><span class="n">manager</span> <span class="o">=</span> <span class="n">BaseManager</span><span class="p">(</span><span class="n">address</span><span class="o">=</span><span class="p">(</span><span class="s1">''</span><span class="p">,</span> <span class="mi">50000</span><span class="p">),</span> <span class="n">authkey</span><span class="o">=</span><span class="sa">b</span><span class="s1">'abc'</span><span class="p">)</span>
|
||
<span class="gp">>>> </span><span class="n">server</span> <span class="o">=</span> <span class="n">manager</span><span class="o">.</span><span class="n">get_server</span><span class="p">()</span>
|
||
<span class="gp">>>> </span><span class="n">server</span><span class="o">.</span><span class="n">serve_forever</span><span class="p">()</span>
|
||
</pre></div>
|
||
</div>
|
||
<p><code class="xref py py-class docutils literal notranslate"><span class="pre">Server</span></code> additionally has an <a class="reference internal" href="#multiprocessing.managers.BaseManager.address" title="multiprocessing.managers.BaseManager.address"><code class="xref py py-attr docutils literal notranslate"><span class="pre">address</span></code></a> attribute.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.managers.BaseManager.connect">
|
||
<code class="descname">connect</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.BaseManager.connect" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Connect a local manager object to a remote manager process:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">multiprocessing.managers</span> <span class="k">import</span> <span class="n">BaseManager</span>
|
||
<span class="gp">>>> </span><span class="n">m</span> <span class="o">=</span> <span class="n">BaseManager</span><span class="p">(</span><span class="n">address</span><span class="o">=</span><span class="p">(</span><span class="s1">'127.0.0.1'</span><span class="p">,</span> <span class="mi">50000</span><span class="p">),</span> <span class="n">authkey</span><span class="o">=</span><span class="sa">b</span><span class="s1">'abc'</span><span class="p">)</span>
|
||
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">connect</span><span class="p">()</span>
|
||
</pre></div>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.managers.BaseManager.shutdown">
|
||
<code class="descname">shutdown</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.BaseManager.shutdown" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Stop the process used by the manager. This is only available if
|
||
<a class="reference internal" href="#multiprocessing.managers.BaseManager.start" title="multiprocessing.managers.BaseManager.start"><code class="xref py py-meth docutils literal notranslate"><span class="pre">start()</span></code></a> has been used to start the server process.</p>
|
||
<p>This can be called multiple times.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.managers.BaseManager.register">
|
||
<code class="descname">register</code><span class="sig-paren">(</span><em>typeid</em><span class="optional">[</span>, <em>callable</em><span class="optional">[</span>, <em>proxytype</em><span class="optional">[</span>, <em>exposed</em><span class="optional">[</span>, <em>method_to_typeid</em><span class="optional">[</span>, <em>create_method</em><span class="optional">]</span><span class="optional">]</span><span class="optional">]</span><span class="optional">]</span><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.BaseManager.register" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>A classmethod which can be used for registering a type or callable with
|
||
the manager class.</p>
|
||
<p><em>typeid</em> is a “type identifier” which is used to identify a particular
|
||
type of shared object. This must be a string.</p>
|
||
<p><em>callable</em> is a callable used for creating objects for this type
|
||
identifier. If a manager instance will be connected to the
|
||
server using the <a class="reference internal" href="#multiprocessing.managers.BaseManager.connect" title="multiprocessing.managers.BaseManager.connect"><code class="xref py py-meth docutils literal notranslate"><span class="pre">connect()</span></code></a> method, or if the
|
||
<em>create_method</em> argument is <code class="docutils literal notranslate"><span class="pre">False</span></code> then this can be left as
|
||
<code class="docutils literal notranslate"><span class="pre">None</span></code>.</p>
|
||
<p><em>proxytype</em> is a subclass of <a class="reference internal" href="#multiprocessing.managers.BaseProxy" title="multiprocessing.managers.BaseProxy"><code class="xref py py-class docutils literal notranslate"><span class="pre">BaseProxy</span></code></a> which is used to create
|
||
proxies for shared objects with this <em>typeid</em>. If <code class="docutils literal notranslate"><span class="pre">None</span></code> then a proxy
|
||
class is created automatically.</p>
|
||
<p><em>exposed</em> is used to specify a sequence of method names which proxies for
|
||
this typeid should be allowed to access using
|
||
<a class="reference internal" href="#multiprocessing.managers.BaseProxy._callmethod" title="multiprocessing.managers.BaseProxy._callmethod"><code class="xref py py-meth docutils literal notranslate"><span class="pre">BaseProxy._callmethod()</span></code></a>. (If <em>exposed</em> is <code class="docutils literal notranslate"><span class="pre">None</span></code> then
|
||
<code class="xref py py-attr docutils literal notranslate"><span class="pre">proxytype._exposed_</span></code> is used instead if it exists.) In the case
|
||
where no exposed list is specified, all “public methods” of the shared
|
||
object will be accessible. (Here a “public method” means any attribute
|
||
which has a <a class="reference internal" href="../reference/datamodel.html#object.__call__" title="object.__call__"><code class="xref py py-meth docutils literal notranslate"><span class="pre">__call__()</span></code></a> method and whose name does not begin
|
||
with <code class="docutils literal notranslate"><span class="pre">'_'</span></code>.)</p>
|
||
<p><em>method_to_typeid</em> is a mapping used to specify the return type of those
|
||
exposed methods which should return a proxy. It maps method names to
|
||
typeid strings. (If <em>method_to_typeid</em> is <code class="docutils literal notranslate"><span class="pre">None</span></code> then
|
||
<code class="xref py py-attr docutils literal notranslate"><span class="pre">proxytype._method_to_typeid_</span></code> is used instead if it exists.) If a
|
||
method’s name is not a key of this mapping or if the mapping is <code class="docutils literal notranslate"><span class="pre">None</span></code>
|
||
then the object returned by the method will be copied by value.</p>
|
||
<p><em>create_method</em> determines whether a method should be created with name
|
||
<em>typeid</em> which can be used to tell the server process to create a new
|
||
shared object and return a proxy for it. By default it is <code class="docutils literal notranslate"><span class="pre">True</span></code>.</p>
|
||
</dd></dl>
|
||
|
||
<p><a class="reference internal" href="#multiprocessing.managers.BaseManager" title="multiprocessing.managers.BaseManager"><code class="xref py py-class docutils literal notranslate"><span class="pre">BaseManager</span></code></a> instances also have one read-only property:</p>
|
||
<dl class="attribute">
|
||
<dt id="multiprocessing.managers.BaseManager.address">
|
||
<code class="descname">address</code><a class="headerlink" href="#multiprocessing.managers.BaseManager.address" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>The address used by the manager.</p>
|
||
</dd></dl>
|
||
|
||
<div class="versionchanged">
|
||
<p><span class="versionmodified changed">Changed in version 3.3: </span>Manager objects support the context management protocol – see
|
||
<a class="reference internal" href="stdtypes.html#typecontextmanager"><span class="std std-ref">Context Manager Types</span></a>. <a class="reference internal" href="stdtypes.html#contextmanager.__enter__" title="contextmanager.__enter__"><code class="xref py py-meth docutils literal notranslate"><span class="pre">__enter__()</span></code></a> starts the
|
||
server process (if it has not already started) and then returns the
|
||
manager object. <a class="reference internal" href="stdtypes.html#contextmanager.__exit__" title="contextmanager.__exit__"><code class="xref py py-meth docutils literal notranslate"><span class="pre">__exit__()</span></code></a> calls <a class="reference internal" href="#multiprocessing.managers.BaseManager.shutdown" title="multiprocessing.managers.BaseManager.shutdown"><code class="xref py py-meth docutils literal notranslate"><span class="pre">shutdown()</span></code></a>.</p>
|
||
<p>In previous versions <a class="reference internal" href="stdtypes.html#contextmanager.__enter__" title="contextmanager.__enter__"><code class="xref py py-meth docutils literal notranslate"><span class="pre">__enter__()</span></code></a> did not start the
|
||
manager’s server process if it was not already started.</p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="class">
|
||
<dt id="multiprocessing.managers.SyncManager">
|
||
<em class="property">class </em><code class="descclassname">multiprocessing.managers.</code><code class="descname">SyncManager</code><a class="headerlink" href="#multiprocessing.managers.SyncManager" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>A subclass of <a class="reference internal" href="#multiprocessing.managers.BaseManager" title="multiprocessing.managers.BaseManager"><code class="xref py py-class docutils literal notranslate"><span class="pre">BaseManager</span></code></a> which can be used for the synchronization
|
||
of processes. Objects of this type are returned by
|
||
<code class="xref py py-func docutils literal notranslate"><span class="pre">multiprocessing.Manager()</span></code>.</p>
|
||
<p>Its methods create and return <a class="reference internal" href="#multiprocessing-proxy-objects"><span class="std std-ref">Proxy Objects</span></a> for a
|
||
number of commonly used data types to be synchronized across processes.
|
||
This notably includes shared lists and dictionaries.</p>
|
||
<dl class="method">
|
||
<dt id="multiprocessing.managers.SyncManager.Barrier">
|
||
<code class="descname">Barrier</code><span class="sig-paren">(</span><em>parties</em><span class="optional">[</span>, <em>action</em><span class="optional">[</span>, <em>timeout</em><span class="optional">]</span><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.SyncManager.Barrier" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Create a shared <a class="reference internal" href="threading.html#threading.Barrier" title="threading.Barrier"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.Barrier</span></code></a> object and return a
|
||
proxy for it.</p>
|
||
<div class="versionadded">
|
||
<p><span class="versionmodified added">New in version 3.3.</span></p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.managers.SyncManager.BoundedSemaphore">
|
||
<code class="descname">BoundedSemaphore</code><span class="sig-paren">(</span><span class="optional">[</span><em>value</em><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.SyncManager.BoundedSemaphore" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Create a shared <a class="reference internal" href="threading.html#threading.BoundedSemaphore" title="threading.BoundedSemaphore"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.BoundedSemaphore</span></code></a> object and return a
|
||
proxy for it.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.managers.SyncManager.Condition">
|
||
<code class="descname">Condition</code><span class="sig-paren">(</span><span class="optional">[</span><em>lock</em><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.SyncManager.Condition" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Create a shared <a class="reference internal" href="threading.html#threading.Condition" title="threading.Condition"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.Condition</span></code></a> object and return a proxy for
|
||
it.</p>
|
||
<p>If <em>lock</em> is supplied then it should be a proxy for a
|
||
<a class="reference internal" href="threading.html#threading.Lock" title="threading.Lock"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.Lock</span></code></a> or <a class="reference internal" href="threading.html#threading.RLock" title="threading.RLock"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.RLock</span></code></a> object.</p>
|
||
<div class="versionchanged">
|
||
<p><span class="versionmodified changed">Changed in version 3.3: </span>The <a class="reference internal" href="threading.html#threading.Condition.wait_for" title="threading.Condition.wait_for"><code class="xref py py-meth docutils literal notranslate"><span class="pre">wait_for()</span></code></a> method was added.</p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.managers.SyncManager.Event">
|
||
<code class="descname">Event</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.SyncManager.Event" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Create a shared <a class="reference internal" href="threading.html#threading.Event" title="threading.Event"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.Event</span></code></a> object and return a proxy for it.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.managers.SyncManager.Lock">
|
||
<code class="descname">Lock</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.SyncManager.Lock" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Create a shared <a class="reference internal" href="threading.html#threading.Lock" title="threading.Lock"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.Lock</span></code></a> object and return a proxy for it.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.managers.SyncManager.Namespace">
|
||
<code class="descname">Namespace</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.SyncManager.Namespace" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Create a shared <a class="reference internal" href="#multiprocessing.managers.Namespace" title="multiprocessing.managers.Namespace"><code class="xref py py-class docutils literal notranslate"><span class="pre">Namespace</span></code></a> object and return a proxy for it.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.managers.SyncManager.Queue">
|
||
<code class="descname">Queue</code><span class="sig-paren">(</span><span class="optional">[</span><em>maxsize</em><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.SyncManager.Queue" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Create a shared <a class="reference internal" href="queue.html#queue.Queue" title="queue.Queue"><code class="xref py py-class docutils literal notranslate"><span class="pre">queue.Queue</span></code></a> object and return a proxy for it.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.managers.SyncManager.RLock">
|
||
<code class="descname">RLock</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.SyncManager.RLock" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Create a shared <a class="reference internal" href="threading.html#threading.RLock" title="threading.RLock"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.RLock</span></code></a> object and return a proxy for it.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.managers.SyncManager.Semaphore">
|
||
<code class="descname">Semaphore</code><span class="sig-paren">(</span><span class="optional">[</span><em>value</em><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.SyncManager.Semaphore" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Create a shared <a class="reference internal" href="threading.html#threading.Semaphore" title="threading.Semaphore"><code class="xref py py-class docutils literal notranslate"><span class="pre">threading.Semaphore</span></code></a> object and return a proxy for
|
||
it.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.managers.SyncManager.Array">
|
||
<code class="descname">Array</code><span class="sig-paren">(</span><em>typecode</em>, <em>sequence</em><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.SyncManager.Array" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Create an array and return a proxy for it.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.managers.SyncManager.Value">
|
||
<code class="descname">Value</code><span class="sig-paren">(</span><em>typecode</em>, <em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.SyncManager.Value" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Create an object with a writable <code class="docutils literal notranslate"><span class="pre">value</span></code> attribute and return a proxy
|
||
for it.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.managers.SyncManager.dict">
|
||
<code class="descname">dict</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.SyncManager.dict" title="Permalink to this definition">¶</a></dt>
|
||
<dt>
|
||
<code class="descname">dict</code><span class="sig-paren">(</span><em>mapping</em><span class="sig-paren">)</span></dt>
|
||
<dt>
|
||
<code class="descname">dict</code><span class="sig-paren">(</span><em>sequence</em><span class="sig-paren">)</span></dt>
|
||
<dd><p>Create a shared <a class="reference internal" href="stdtypes.html#dict" title="dict"><code class="xref py py-class docutils literal notranslate"><span class="pre">dict</span></code></a> object and return a proxy for it.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.managers.SyncManager.list">
|
||
<code class="descname">list</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.SyncManager.list" title="Permalink to this definition">¶</a></dt>
|
||
<dt>
|
||
<code class="descname">list</code><span class="sig-paren">(</span><em>sequence</em><span class="sig-paren">)</span></dt>
|
||
<dd><p>Create a shared <a class="reference internal" href="stdtypes.html#list" title="list"><code class="xref py py-class docutils literal notranslate"><span class="pre">list</span></code></a> object and return a proxy for it.</p>
|
||
</dd></dl>
|
||
|
||
<div class="versionchanged">
|
||
<p><span class="versionmodified changed">Changed in version 3.6: </span>Shared objects are capable of being nested. For example, a shared
|
||
container object such as a shared list can contain other shared objects
|
||
which will all be managed and synchronized by the <a class="reference internal" href="#multiprocessing.managers.SyncManager" title="multiprocessing.managers.SyncManager"><code class="xref py py-class docutils literal notranslate"><span class="pre">SyncManager</span></code></a>.</p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="class">
|
||
<dt id="multiprocessing.managers.Namespace">
|
||
<em class="property">class </em><code class="descclassname">multiprocessing.managers.</code><code class="descname">Namespace</code><a class="headerlink" href="#multiprocessing.managers.Namespace" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>A type that can register with <a class="reference internal" href="#multiprocessing.managers.SyncManager" title="multiprocessing.managers.SyncManager"><code class="xref py py-class docutils literal notranslate"><span class="pre">SyncManager</span></code></a>.</p>
|
||
<p>A namespace object has no public methods, but does have writable attributes.
|
||
Its representation shows the values of its attributes.</p>
|
||
<p>However, when using a proxy for a namespace object, an attribute beginning
|
||
with <code class="docutils literal notranslate"><span class="pre">'_'</span></code> will be an attribute of the proxy and not an attribute of the
|
||
referent:</p>
|
||
<div class="highlight-pycon3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">manager</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Manager</span><span class="p">()</span>
|
||
<span class="gp">>>> </span><span class="n">Global</span> <span class="o">=</span> <span class="n">manager</span><span class="o">.</span><span class="n">Namespace</span><span class="p">()</span>
|
||
<span class="gp">>>> </span><span class="n">Global</span><span class="o">.</span><span class="n">x</span> <span class="o">=</span> <span class="mi">10</span>
|
||
<span class="gp">>>> </span><span class="n">Global</span><span class="o">.</span><span class="n">y</span> <span class="o">=</span> <span class="s1">'hello'</span>
|
||
<span class="gp">>>> </span><span class="n">Global</span><span class="o">.</span><span class="n">_z</span> <span class="o">=</span> <span class="mf">12.3</span> <span class="c1"># this is an attribute of the proxy</span>
|
||
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">Global</span><span class="p">)</span>
|
||
<span class="go">Namespace(x=10, y='hello')</span>
|
||
</pre></div>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<div class="section" id="customized-managers">
|
||
<h4>Customized managers<a class="headerlink" href="#customized-managers" title="Permalink to this headline">¶</a></h4>
|
||
<p>To create one’s own manager, one creates a subclass of <a class="reference internal" href="#multiprocessing.managers.BaseManager" title="multiprocessing.managers.BaseManager"><code class="xref py py-class docutils literal notranslate"><span class="pre">BaseManager</span></code></a> and
|
||
uses the <a class="reference internal" href="#multiprocessing.managers.BaseManager.register" title="multiprocessing.managers.BaseManager.register"><code class="xref py py-meth docutils literal notranslate"><span class="pre">register()</span></code></a> classmethod to register new types or
|
||
callables with the manager class. For example:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing.managers</span> <span class="k">import</span> <span class="n">BaseManager</span>
|
||
|
||
<span class="k">class</span> <span class="nc">MathsClass</span><span class="p">:</span>
|
||
<span class="k">def</span> <span class="nf">add</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span>
|
||
<span class="k">def</span> <span class="nf">mul</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="n">x</span> <span class="o">*</span> <span class="n">y</span>
|
||
|
||
<span class="k">class</span> <span class="nc">MyManager</span><span class="p">(</span><span class="n">BaseManager</span><span class="p">):</span>
|
||
<span class="k">pass</span>
|
||
|
||
<span class="n">MyManager</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="s1">'Maths'</span><span class="p">,</span> <span class="n">MathsClass</span><span class="p">)</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="k">with</span> <span class="n">MyManager</span><span class="p">()</span> <span class="k">as</span> <span class="n">manager</span><span class="p">:</span>
|
||
<span class="n">maths</span> <span class="o">=</span> <span class="n">manager</span><span class="o">.</span><span class="n">Maths</span><span class="p">()</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">maths</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="mi">3</span><span class="p">))</span> <span class="c1"># prints 7</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">maths</span><span class="o">.</span><span class="n">mul</span><span class="p">(</span><span class="mi">7</span><span class="p">,</span> <span class="mi">8</span><span class="p">))</span> <span class="c1"># prints 56</span>
|
||
</pre></div>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="using-a-remote-manager">
|
||
<h4>Using a remote manager<a class="headerlink" href="#using-a-remote-manager" title="Permalink to this headline">¶</a></h4>
|
||
<p>It is possible to run a manager server on one machine and have clients use it
|
||
from other machines (assuming that the firewalls involved allow it).</p>
|
||
<p>Running the following commands creates a server for a single shared queue which
|
||
remote clients can access:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">multiprocessing.managers</span> <span class="k">import</span> <span class="n">BaseManager</span>
|
||
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">queue</span> <span class="k">import</span> <span class="n">Queue</span>
|
||
<span class="gp">>>> </span><span class="n">queue</span> <span class="o">=</span> <span class="n">Queue</span><span class="p">()</span>
|
||
<span class="gp">>>> </span><span class="k">class</span> <span class="nc">QueueManager</span><span class="p">(</span><span class="n">BaseManager</span><span class="p">):</span> <span class="k">pass</span>
|
||
<span class="gp">>>> </span><span class="n">QueueManager</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="s1">'get_queue'</span><span class="p">,</span> <span class="n">callable</span><span class="o">=</span><span class="k">lambda</span><span class="p">:</span><span class="n">queue</span><span class="p">)</span>
|
||
<span class="gp">>>> </span><span class="n">m</span> <span class="o">=</span> <span class="n">QueueManager</span><span class="p">(</span><span class="n">address</span><span class="o">=</span><span class="p">(</span><span class="s1">''</span><span class="p">,</span> <span class="mi">50000</span><span class="p">),</span> <span class="n">authkey</span><span class="o">=</span><span class="sa">b</span><span class="s1">'abracadabra'</span><span class="p">)</span>
|
||
<span class="gp">>>> </span><span class="n">s</span> <span class="o">=</span> <span class="n">m</span><span class="o">.</span><span class="n">get_server</span><span class="p">()</span>
|
||
<span class="gp">>>> </span><span class="n">s</span><span class="o">.</span><span class="n">serve_forever</span><span class="p">()</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>One client can access the server as follows:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">multiprocessing.managers</span> <span class="k">import</span> <span class="n">BaseManager</span>
|
||
<span class="gp">>>> </span><span class="k">class</span> <span class="nc">QueueManager</span><span class="p">(</span><span class="n">BaseManager</span><span class="p">):</span> <span class="k">pass</span>
|
||
<span class="gp">>>> </span><span class="n">QueueManager</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="s1">'get_queue'</span><span class="p">)</span>
|
||
<span class="gp">>>> </span><span class="n">m</span> <span class="o">=</span> <span class="n">QueueManager</span><span class="p">(</span><span class="n">address</span><span class="o">=</span><span class="p">(</span><span class="s1">'foo.bar.org'</span><span class="p">,</span> <span class="mi">50000</span><span class="p">),</span> <span class="n">authkey</span><span class="o">=</span><span class="sa">b</span><span class="s1">'abracadabra'</span><span class="p">)</span>
|
||
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">connect</span><span class="p">()</span>
|
||
<span class="gp">>>> </span><span class="n">queue</span> <span class="o">=</span> <span class="n">m</span><span class="o">.</span><span class="n">get_queue</span><span class="p">()</span>
|
||
<span class="gp">>>> </span><span class="n">queue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">)</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>Another client can also use it:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">multiprocessing.managers</span> <span class="k">import</span> <span class="n">BaseManager</span>
|
||
<span class="gp">>>> </span><span class="k">class</span> <span class="nc">QueueManager</span><span class="p">(</span><span class="n">BaseManager</span><span class="p">):</span> <span class="k">pass</span>
|
||
<span class="gp">>>> </span><span class="n">QueueManager</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="s1">'get_queue'</span><span class="p">)</span>
|
||
<span class="gp">>>> </span><span class="n">m</span> <span class="o">=</span> <span class="n">QueueManager</span><span class="p">(</span><span class="n">address</span><span class="o">=</span><span class="p">(</span><span class="s1">'foo.bar.org'</span><span class="p">,</span> <span class="mi">50000</span><span class="p">),</span> <span class="n">authkey</span><span class="o">=</span><span class="sa">b</span><span class="s1">'abracadabra'</span><span class="p">)</span>
|
||
<span class="gp">>>> </span><span class="n">m</span><span class="o">.</span><span class="n">connect</span><span class="p">()</span>
|
||
<span class="gp">>>> </span><span class="n">queue</span> <span class="o">=</span> <span class="n">m</span><span class="o">.</span><span class="n">get_queue</span><span class="p">()</span>
|
||
<span class="gp">>>> </span><span class="n">queue</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
|
||
<span class="go">'hello'</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>Local processes can also access that queue, using the code from above on the
|
||
client to access it remotely:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Process</span><span class="p">,</span> <span class="n">Queue</span>
|
||
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">multiprocessing.managers</span> <span class="k">import</span> <span class="n">BaseManager</span>
|
||
<span class="gp">>>> </span><span class="k">class</span> <span class="nc">Worker</span><span class="p">(</span><span class="n">Process</span><span class="p">):</span>
|
||
<span class="gp">... </span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">q</span><span class="p">):</span>
|
||
<span class="gp">... </span> <span class="bp">self</span><span class="o">.</span><span class="n">q</span> <span class="o">=</span> <span class="n">q</span>
|
||
<span class="gp">... </span> <span class="nb">super</span><span class="p">(</span><span class="n">Worker</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
|
||
<span class="gp">... </span> <span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="gp">... </span> <span class="bp">self</span><span class="o">.</span><span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'local hello'</span><span class="p">)</span>
|
||
<span class="gp">...</span>
|
||
<span class="gp">>>> </span><span class="n">queue</span> <span class="o">=</span> <span class="n">Queue</span><span class="p">()</span>
|
||
<span class="gp">>>> </span><span class="n">w</span> <span class="o">=</span> <span class="n">Worker</span><span class="p">(</span><span class="n">queue</span><span class="p">)</span>
|
||
<span class="gp">>>> </span><span class="n">w</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
|
||
<span class="gp">>>> </span><span class="k">class</span> <span class="nc">QueueManager</span><span class="p">(</span><span class="n">BaseManager</span><span class="p">):</span> <span class="k">pass</span>
|
||
<span class="gp">...</span>
|
||
<span class="gp">>>> </span><span class="n">QueueManager</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="s1">'get_queue'</span><span class="p">,</span> <span class="n">callable</span><span class="o">=</span><span class="k">lambda</span><span class="p">:</span> <span class="n">queue</span><span class="p">)</span>
|
||
<span class="gp">>>> </span><span class="n">m</span> <span class="o">=</span> <span class="n">QueueManager</span><span class="p">(</span><span class="n">address</span><span class="o">=</span><span class="p">(</span><span class="s1">''</span><span class="p">,</span> <span class="mi">50000</span><span class="p">),</span> <span class="n">authkey</span><span class="o">=</span><span class="sa">b</span><span class="s1">'abracadabra'</span><span class="p">)</span>
|
||
<span class="gp">>>> </span><span class="n">s</span> <span class="o">=</span> <span class="n">m</span><span class="o">.</span><span class="n">get_server</span><span class="p">()</span>
|
||
<span class="gp">>>> </span><span class="n">s</span><span class="o">.</span><span class="n">serve_forever</span><span class="p">()</span>
|
||
</pre></div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="proxy-objects">
|
||
<span id="multiprocessing-proxy-objects"></span><h3>Proxy Objects<a class="headerlink" href="#proxy-objects" title="Permalink to this headline">¶</a></h3>
|
||
<p>A proxy is an object which <em>refers</em> to a shared object which lives (presumably)
|
||
in a different process. The shared object is said to be the <em>referent</em> of the
|
||
proxy. Multiple proxy objects may have the same referent.</p>
|
||
<p>A proxy object has methods which invoke corresponding methods of its referent
|
||
(although not every method of the referent will necessarily be available through
|
||
the proxy). In this way, a proxy can be used just like its referent can:</p>
|
||
<div class="highlight-pycon3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Manager</span>
|
||
<span class="gp">>>> </span><span class="n">manager</span> <span class="o">=</span> <span class="n">Manager</span><span class="p">()</span>
|
||
<span class="gp">>>> </span><span class="n">l</span> <span class="o">=</span> <span class="n">manager</span><span class="o">.</span><span class="n">list</span><span class="p">([</span><span class="n">i</span><span class="o">*</span><span class="n">i</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)])</span>
|
||
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">l</span><span class="p">)</span>
|
||
<span class="go">[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]</span>
|
||
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="nb">repr</span><span class="p">(</span><span class="n">l</span><span class="p">))</span>
|
||
<span class="go"><ListProxy object, typeid 'list' at 0x...></span>
|
||
<span class="gp">>>> </span><span class="n">l</span><span class="p">[</span><span class="mi">4</span><span class="p">]</span>
|
||
<span class="go">16</span>
|
||
<span class="gp">>>> </span><span class="n">l</span><span class="p">[</span><span class="mi">2</span><span class="p">:</span><span class="mi">5</span><span class="p">]</span>
|
||
<span class="go">[4, 9, 16]</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>Notice that applying <a class="reference internal" href="stdtypes.html#str" title="str"><code class="xref py py-func docutils literal notranslate"><span class="pre">str()</span></code></a> to a proxy will return the representation of
|
||
the referent, whereas applying <a class="reference internal" href="functions.html#repr" title="repr"><code class="xref py py-func docutils literal notranslate"><span class="pre">repr()</span></code></a> will return the representation of
|
||
the proxy.</p>
|
||
<p>An important feature of proxy objects is that they are picklable so they can be
|
||
passed between processes. As such, a referent can contain
|
||
<a class="reference internal" href="#multiprocessing-proxy-objects"><span class="std std-ref">Proxy Objects</span></a>. This permits nesting of these managed
|
||
lists, dicts, and other <a class="reference internal" href="#multiprocessing-proxy-objects"><span class="std std-ref">Proxy Objects</span></a>:</p>
|
||
<div class="highlight-pycon3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">a</span> <span class="o">=</span> <span class="n">manager</span><span class="o">.</span><span class="n">list</span><span class="p">()</span>
|
||
<span class="gp">>>> </span><span class="n">b</span> <span class="o">=</span> <span class="n">manager</span><span class="o">.</span><span class="n">list</span><span class="p">()</span>
|
||
<span class="gp">>>> </span><span class="n">a</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="c1"># referent of a now contains referent of b</span>
|
||
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span>
|
||
<span class="go">[<ListProxy object, typeid 'list' at ...>] []</span>
|
||
<span class="gp">>>> </span><span class="n">b</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">)</span>
|
||
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">a</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">b</span><span class="p">)</span>
|
||
<span class="go">['hello'] ['hello']</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>Similarly, dict and list proxies may be nested inside one another:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">l_outer</span> <span class="o">=</span> <span class="n">manager</span><span class="o">.</span><span class="n">list</span><span class="p">([</span> <span class="n">manager</span><span class="o">.</span><span class="n">dict</span><span class="p">()</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="p">])</span>
|
||
<span class="gp">>>> </span><span class="n">d_first_inner</span> <span class="o">=</span> <span class="n">l_outer</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
|
||
<span class="gp">>>> </span><span class="n">d_first_inner</span><span class="p">[</span><span class="s1">'a'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span>
|
||
<span class="gp">>>> </span><span class="n">d_first_inner</span><span class="p">[</span><span class="s1">'b'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">2</span>
|
||
<span class="gp">>>> </span><span class="n">l_outer</span><span class="p">[</span><span class="mi">1</span><span class="p">][</span><span class="s1">'c'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">3</span>
|
||
<span class="gp">>>> </span><span class="n">l_outer</span><span class="p">[</span><span class="mi">1</span><span class="p">][</span><span class="s1">'z'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">26</span>
|
||
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">l_outer</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
|
||
<span class="go">{'a': 1, 'b': 2}</span>
|
||
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">l_outer</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
|
||
<span class="go">{'c': 3, 'z': 26}</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>If standard (non-proxy) <a class="reference internal" href="stdtypes.html#list" title="list"><code class="xref py py-class docutils literal notranslate"><span class="pre">list</span></code></a> or <a class="reference internal" href="stdtypes.html#dict" title="dict"><code class="xref py py-class docutils literal notranslate"><span class="pre">dict</span></code></a> objects are contained
|
||
in a referent, modifications to those mutable values will not be propagated
|
||
through the manager because the proxy has no way of knowing when the values
|
||
contained within are modified. However, storing a value in a container proxy
|
||
(which triggers a <code class="docutils literal notranslate"><span class="pre">__setitem__</span></code> on the proxy object) does propagate through
|
||
the manager and so to effectively modify such an item, one could re-assign the
|
||
modified value to the container proxy:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># create a list proxy and append a mutable object (a dictionary)</span>
|
||
<span class="n">lproxy</span> <span class="o">=</span> <span class="n">manager</span><span class="o">.</span><span class="n">list</span><span class="p">()</span>
|
||
<span class="n">lproxy</span><span class="o">.</span><span class="n">append</span><span class="p">({})</span>
|
||
<span class="c1"># now mutate the dictionary</span>
|
||
<span class="n">d</span> <span class="o">=</span> <span class="n">lproxy</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
|
||
<span class="n">d</span><span class="p">[</span><span class="s1">'a'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span>
|
||
<span class="n">d</span><span class="p">[</span><span class="s1">'b'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">2</span>
|
||
<span class="c1"># at this point, the changes to d are not yet synced, but by</span>
|
||
<span class="c1"># updating the dictionary, the proxy is notified of the change</span>
|
||
<span class="n">lproxy</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="n">d</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>This approach is perhaps less convenient than employing nested
|
||
<a class="reference internal" href="#multiprocessing-proxy-objects"><span class="std std-ref">Proxy Objects</span></a> for most use cases but also
|
||
demonstrates a level of control over the synchronization.</p>
|
||
<div class="admonition note">
|
||
<p class="admonition-title">Note</p>
|
||
<p>The proxy types in <a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a> do nothing to support comparisons
|
||
by value. So, for instance, we have:</p>
|
||
<div class="highlight-pycon3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">manager</span><span class="o">.</span><span class="n">list</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">])</span> <span class="o">==</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">]</span>
|
||
<span class="go">False</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>One should just use a copy of the referent instead when making comparisons.</p>
|
||
</div>
|
||
<dl class="class">
|
||
<dt id="multiprocessing.managers.BaseProxy">
|
||
<em class="property">class </em><code class="descclassname">multiprocessing.managers.</code><code class="descname">BaseProxy</code><a class="headerlink" href="#multiprocessing.managers.BaseProxy" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Proxy objects are instances of subclasses of <a class="reference internal" href="#multiprocessing.managers.BaseProxy" title="multiprocessing.managers.BaseProxy"><code class="xref py py-class docutils literal notranslate"><span class="pre">BaseProxy</span></code></a>.</p>
|
||
<dl class="method">
|
||
<dt id="multiprocessing.managers.BaseProxy._callmethod">
|
||
<code class="descname">_callmethod</code><span class="sig-paren">(</span><em>methodname</em><span class="optional">[</span>, <em>args</em><span class="optional">[</span>, <em>kwds</em><span class="optional">]</span><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.BaseProxy._callmethod" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Call and return the result of a method of the proxy’s referent.</p>
|
||
<p>If <code class="docutils literal notranslate"><span class="pre">proxy</span></code> is a proxy whose referent is <code class="docutils literal notranslate"><span class="pre">obj</span></code> then the expression</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="n">proxy</span><span class="o">.</span><span class="n">_callmethod</span><span class="p">(</span><span class="n">methodname</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="n">kwds</span><span class="p">)</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>will evaluate the expression</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="nb">getattr</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span> <span class="n">methodname</span><span class="p">)(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwds</span><span class="p">)</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>in the manager’s process.</p>
|
||
<p>The returned value will be a copy of the result of the call or a proxy to
|
||
a new shared object – see documentation for the <em>method_to_typeid</em>
|
||
argument of <a class="reference internal" href="#multiprocessing.managers.BaseManager.register" title="multiprocessing.managers.BaseManager.register"><code class="xref py py-meth docutils literal notranslate"><span class="pre">BaseManager.register()</span></code></a>.</p>
|
||
<p>If an exception is raised by the call, then is re-raised by
|
||
<a class="reference internal" href="#multiprocessing.managers.BaseProxy._callmethod" title="multiprocessing.managers.BaseProxy._callmethod"><code class="xref py py-meth docutils literal notranslate"><span class="pre">_callmethod()</span></code></a>. If some other exception is raised in the manager’s
|
||
process then this is converted into a <code class="xref py py-exc docutils literal notranslate"><span class="pre">RemoteError</span></code> exception and is
|
||
raised by <a class="reference internal" href="#multiprocessing.managers.BaseProxy._callmethod" title="multiprocessing.managers.BaseProxy._callmethod"><code class="xref py py-meth docutils literal notranslate"><span class="pre">_callmethod()</span></code></a>.</p>
|
||
<p>Note in particular that an exception will be raised if <em>methodname</em> has
|
||
not been <em>exposed</em>.</p>
|
||
<p>An example of the usage of <a class="reference internal" href="#multiprocessing.managers.BaseProxy._callmethod" title="multiprocessing.managers.BaseProxy._callmethod"><code class="xref py py-meth docutils literal notranslate"><span class="pre">_callmethod()</span></code></a>:</p>
|
||
<div class="highlight-pycon3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">l</span> <span class="o">=</span> <span class="n">manager</span><span class="o">.</span><span class="n">list</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">))</span>
|
||
<span class="gp">>>> </span><span class="n">l</span><span class="o">.</span><span class="n">_callmethod</span><span class="p">(</span><span class="s1">'__len__'</span><span class="p">)</span>
|
||
<span class="go">10</span>
|
||
<span class="gp">>>> </span><span class="n">l</span><span class="o">.</span><span class="n">_callmethod</span><span class="p">(</span><span class="s1">'__getitem__'</span><span class="p">,</span> <span class="p">(</span><span class="nb">slice</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">7</span><span class="p">),))</span> <span class="c1"># equivalent to l[2:7]</span>
|
||
<span class="go">[2, 3, 4, 5, 6]</span>
|
||
<span class="gp">>>> </span><span class="n">l</span><span class="o">.</span><span class="n">_callmethod</span><span class="p">(</span><span class="s1">'__getitem__'</span><span class="p">,</span> <span class="p">(</span><span class="mi">20</span><span class="p">,))</span> <span class="c1"># equivalent to l[20]</span>
|
||
<span class="gt">Traceback (most recent call last):</span>
|
||
<span class="c">...</span>
|
||
<span class="gr">IndexError</span>: <span class="n">list index out of range</span>
|
||
</pre></div>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.managers.BaseProxy._getvalue">
|
||
<code class="descname">_getvalue</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.BaseProxy._getvalue" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return a copy of the referent.</p>
|
||
<p>If the referent is unpicklable then this will raise an exception.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.managers.BaseProxy.__repr__">
|
||
<code class="descname">__repr__</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.BaseProxy.__repr__" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return a representation of the proxy object.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.managers.BaseProxy.__str__">
|
||
<code class="descname">__str__</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.managers.BaseProxy.__str__" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return the representation of the referent.</p>
|
||
</dd></dl>
|
||
|
||
</dd></dl>
|
||
|
||
<div class="section" id="cleanup">
|
||
<h4>Cleanup<a class="headerlink" href="#cleanup" title="Permalink to this headline">¶</a></h4>
|
||
<p>A proxy object uses a weakref callback so that when it gets garbage collected it
|
||
deregisters itself from the manager which owns its referent.</p>
|
||
<p>A shared object gets deleted from the manager process when there are no longer
|
||
any proxies referring to it.</p>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="module-multiprocessing.pool">
|
||
<span id="process-pools"></span><h3>Process Pools<a class="headerlink" href="#module-multiprocessing.pool" title="Permalink to this headline">¶</a></h3>
|
||
<p>One can create a pool of processes which will carry out tasks submitted to it
|
||
with the <a class="reference internal" href="#multiprocessing.pool.Pool" title="multiprocessing.pool.Pool"><code class="xref py py-class docutils literal notranslate"><span class="pre">Pool</span></code></a> class.</p>
|
||
<dl class="class">
|
||
<dt id="multiprocessing.pool.Pool">
|
||
<em class="property">class </em><code class="descclassname">multiprocessing.pool.</code><code class="descname">Pool</code><span class="sig-paren">(</span><span class="optional">[</span><em>processes</em><span class="optional">[</span>, <em>initializer</em><span class="optional">[</span>, <em>initargs</em><span class="optional">[</span>, <em>maxtasksperchild</em><span class="optional">[</span>, <em>context</em><span class="optional">]</span><span class="optional">]</span><span class="optional">]</span><span class="optional">]</span><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.pool.Pool" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>A process pool object which controls a pool of worker processes to which jobs
|
||
can be submitted. It supports asynchronous results with timeouts and
|
||
callbacks and has a parallel map implementation.</p>
|
||
<p><em>processes</em> is the number of worker processes to use. If <em>processes</em> is
|
||
<code class="docutils literal notranslate"><span class="pre">None</span></code> then the number returned by <a class="reference internal" href="os.html#os.cpu_count" title="os.cpu_count"><code class="xref py py-func docutils literal notranslate"><span class="pre">os.cpu_count()</span></code></a> is used.</p>
|
||
<p>If <em>initializer</em> is not <code class="docutils literal notranslate"><span class="pre">None</span></code> then each worker process will call
|
||
<code class="docutils literal notranslate"><span class="pre">initializer(*initargs)</span></code> when it starts.</p>
|
||
<p><em>maxtasksperchild</em> is the number of tasks a worker process can complete
|
||
before it will exit and be replaced with a fresh worker process, to enable
|
||
unused resources to be freed. The default <em>maxtasksperchild</em> is <code class="docutils literal notranslate"><span class="pre">None</span></code>, which
|
||
means worker processes will live as long as the pool.</p>
|
||
<p><em>context</em> can be used to specify the context used for starting
|
||
the worker processes. Usually a pool is created using the
|
||
function <code class="xref py py-func docutils literal notranslate"><span class="pre">multiprocessing.Pool()</span></code> or the <a class="reference internal" href="#multiprocessing.pool.Pool" title="multiprocessing.pool.Pool"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Pool()</span></code></a> method
|
||
of a context object. In both cases <em>context</em> is set
|
||
appropriately.</p>
|
||
<p>Note that the methods of the pool object should only be called by
|
||
the process which created the pool.</p>
|
||
<div class="versionadded">
|
||
<p><span class="versionmodified added">New in version 3.2: </span><em>maxtasksperchild</em></p>
|
||
</div>
|
||
<div class="versionadded">
|
||
<p><span class="versionmodified added">New in version 3.4: </span><em>context</em></p>
|
||
</div>
|
||
<div class="admonition note">
|
||
<p class="admonition-title">Note</p>
|
||
<p>Worker processes within a <a class="reference internal" href="#multiprocessing.pool.Pool" title="multiprocessing.pool.Pool"><code class="xref py py-class docutils literal notranslate"><span class="pre">Pool</span></code></a> typically live for the complete
|
||
duration of the Pool’s work queue. A frequent pattern found in other
|
||
systems (such as Apache, mod_wsgi, etc) to free resources held by
|
||
workers is to allow a worker within a pool to complete only a set
|
||
amount of work before being exiting, being cleaned up and a new
|
||
process spawned to replace the old one. The <em>maxtasksperchild</em>
|
||
argument to the <a class="reference internal" href="#multiprocessing.pool.Pool" title="multiprocessing.pool.Pool"><code class="xref py py-class docutils literal notranslate"><span class="pre">Pool</span></code></a> exposes this ability to the end user.</p>
|
||
</div>
|
||
<dl class="method">
|
||
<dt id="multiprocessing.pool.Pool.apply">
|
||
<code class="descname">apply</code><span class="sig-paren">(</span><em>func</em><span class="optional">[</span>, <em>args</em><span class="optional">[</span>, <em>kwds</em><span class="optional">]</span><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.pool.Pool.apply" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Call <em>func</em> with arguments <em>args</em> and keyword arguments <em>kwds</em>. It blocks
|
||
until the result is ready. Given this blocks, <a class="reference internal" href="#multiprocessing.pool.Pool.apply_async" title="multiprocessing.pool.Pool.apply_async"><code class="xref py py-meth docutils literal notranslate"><span class="pre">apply_async()</span></code></a> is
|
||
better suited for performing work in parallel. Additionally, <em>func</em>
|
||
is only executed in one of the workers of the pool.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.pool.Pool.apply_async">
|
||
<code class="descname">apply_async</code><span class="sig-paren">(</span><em>func</em><span class="optional">[</span>, <em>args</em><span class="optional">[</span>, <em>kwds</em><span class="optional">[</span>, <em>callback</em><span class="optional">[</span>, <em>error_callback</em><span class="optional">]</span><span class="optional">]</span><span class="optional">]</span><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.pool.Pool.apply_async" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>A variant of the <a class="reference internal" href="#multiprocessing.pool.Pool.apply" title="multiprocessing.pool.Pool.apply"><code class="xref py py-meth docutils literal notranslate"><span class="pre">apply()</span></code></a> method which returns a result object.</p>
|
||
<p>If <em>callback</em> is specified then it should be a callable which accepts a
|
||
single argument. When the result becomes ready <em>callback</em> is applied to
|
||
it, that is unless the call failed, in which case the <em>error_callback</em>
|
||
is applied instead.</p>
|
||
<p>If <em>error_callback</em> is specified then it should be a callable which
|
||
accepts a single argument. If the target function fails, then
|
||
the <em>error_callback</em> is called with the exception instance.</p>
|
||
<p>Callbacks should complete immediately since otherwise the thread which
|
||
handles the results will get blocked.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.pool.Pool.map">
|
||
<code class="descname">map</code><span class="sig-paren">(</span><em>func</em>, <em>iterable</em><span class="optional">[</span>, <em>chunksize</em><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.pool.Pool.map" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>A parallel equivalent of the <a class="reference internal" href="functions.html#map" title="map"><code class="xref py py-func docutils literal notranslate"><span class="pre">map()</span></code></a> built-in function (it supports only
|
||
one <em>iterable</em> argument though). It blocks until the result is ready.</p>
|
||
<p>This method chops the iterable into a number of chunks which it submits to
|
||
the process pool as separate tasks. The (approximate) size of these
|
||
chunks can be specified by setting <em>chunksize</em> to a positive integer.</p>
|
||
<p>Note that it may cause high memory usage for very long iterables. Consider
|
||
using <a class="reference internal" href="#multiprocessing.pool.Pool.imap" title="multiprocessing.pool.Pool.imap"><code class="xref py py-meth docutils literal notranslate"><span class="pre">imap()</span></code></a> or <a class="reference internal" href="#multiprocessing.pool.Pool.imap_unordered" title="multiprocessing.pool.Pool.imap_unordered"><code class="xref py py-meth docutils literal notranslate"><span class="pre">imap_unordered()</span></code></a> with explicit <em>chunksize</em>
|
||
option for better efficiency.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.pool.Pool.map_async">
|
||
<code class="descname">map_async</code><span class="sig-paren">(</span><em>func</em>, <em>iterable</em><span class="optional">[</span>, <em>chunksize</em><span class="optional">[</span>, <em>callback</em><span class="optional">[</span>, <em>error_callback</em><span class="optional">]</span><span class="optional">]</span><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.pool.Pool.map_async" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>A variant of the <a class="reference internal" href="#multiprocessing.pool.Pool.map" title="multiprocessing.pool.Pool.map"><code class="xref py py-meth docutils literal notranslate"><span class="pre">map()</span></code></a> method which returns a result object.</p>
|
||
<p>If <em>callback</em> is specified then it should be a callable which accepts a
|
||
single argument. When the result becomes ready <em>callback</em> is applied to
|
||
it, that is unless the call failed, in which case the <em>error_callback</em>
|
||
is applied instead.</p>
|
||
<p>If <em>error_callback</em> is specified then it should be a callable which
|
||
accepts a single argument. If the target function fails, then
|
||
the <em>error_callback</em> is called with the exception instance.</p>
|
||
<p>Callbacks should complete immediately since otherwise the thread which
|
||
handles the results will get blocked.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.pool.Pool.imap">
|
||
<code class="descname">imap</code><span class="sig-paren">(</span><em>func</em>, <em>iterable</em><span class="optional">[</span>, <em>chunksize</em><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.pool.Pool.imap" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>A lazier version of <a class="reference internal" href="#multiprocessing.pool.Pool.map" title="multiprocessing.pool.Pool.map"><code class="xref py py-meth docutils literal notranslate"><span class="pre">map()</span></code></a>.</p>
|
||
<p>The <em>chunksize</em> argument is the same as the one used by the <a class="reference internal" href="#multiprocessing.pool.Pool.map" title="multiprocessing.pool.Pool.map"><code class="xref py py-meth docutils literal notranslate"><span class="pre">map()</span></code></a>
|
||
method. For very long iterables using a large value for <em>chunksize</em> can
|
||
make the job complete <strong>much</strong> faster than using the default value of
|
||
<code class="docutils literal notranslate"><span class="pre">1</span></code>.</p>
|
||
<p>Also if <em>chunksize</em> is <code class="docutils literal notranslate"><span class="pre">1</span></code> then the <code class="xref py py-meth docutils literal notranslate"><span class="pre">next()</span></code> method of the iterator
|
||
returned by the <a class="reference internal" href="#multiprocessing.pool.Pool.imap" title="multiprocessing.pool.Pool.imap"><code class="xref py py-meth docutils literal notranslate"><span class="pre">imap()</span></code></a> method has an optional <em>timeout</em> parameter:
|
||
<code class="docutils literal notranslate"><span class="pre">next(timeout)</span></code> will raise <a class="reference internal" href="#multiprocessing.TimeoutError" title="multiprocessing.TimeoutError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">multiprocessing.TimeoutError</span></code></a> if the
|
||
result cannot be returned within <em>timeout</em> seconds.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.pool.Pool.imap_unordered">
|
||
<code class="descname">imap_unordered</code><span class="sig-paren">(</span><em>func</em>, <em>iterable</em><span class="optional">[</span>, <em>chunksize</em><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.pool.Pool.imap_unordered" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>The same as <a class="reference internal" href="#multiprocessing.pool.Pool.imap" title="multiprocessing.pool.Pool.imap"><code class="xref py py-meth docutils literal notranslate"><span class="pre">imap()</span></code></a> except that the ordering of the results from the
|
||
returned iterator should be considered arbitrary. (Only when there is
|
||
only one worker process is the order guaranteed to be “correct”.)</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.pool.Pool.starmap">
|
||
<code class="descname">starmap</code><span class="sig-paren">(</span><em>func</em>, <em>iterable</em><span class="optional">[</span>, <em>chunksize</em><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.pool.Pool.starmap" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Like <a class="reference internal" href="functions.html#map" title="map"><code class="xref py py-meth docutils literal notranslate"><span class="pre">map()</span></code></a> except that the elements of the <em>iterable</em> are expected
|
||
to be iterables that are unpacked as arguments.</p>
|
||
<p>Hence an <em>iterable</em> of <code class="docutils literal notranslate"><span class="pre">[(1,2),</span> <span class="pre">(3,</span> <span class="pre">4)]</span></code> results in <code class="docutils literal notranslate"><span class="pre">[func(1,2),</span>
|
||
<span class="pre">func(3,4)]</span></code>.</p>
|
||
<div class="versionadded">
|
||
<p><span class="versionmodified added">New in version 3.3.</span></p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.pool.Pool.starmap_async">
|
||
<code class="descname">starmap_async</code><span class="sig-paren">(</span><em>func</em>, <em>iterable</em><span class="optional">[</span>, <em>chunksize</em><span class="optional">[</span>, <em>callback</em><span class="optional">[</span>, <em>error_callback</em><span class="optional">]</span><span class="optional">]</span><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.pool.Pool.starmap_async" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>A combination of <a class="reference internal" href="#multiprocessing.pool.Pool.starmap" title="multiprocessing.pool.Pool.starmap"><code class="xref py py-meth docutils literal notranslate"><span class="pre">starmap()</span></code></a> and <a class="reference internal" href="#multiprocessing.pool.Pool.map_async" title="multiprocessing.pool.Pool.map_async"><code class="xref py py-meth docutils literal notranslate"><span class="pre">map_async()</span></code></a> that iterates over
|
||
<em>iterable</em> of iterables and calls <em>func</em> with the iterables unpacked.
|
||
Returns a result object.</p>
|
||
<div class="versionadded">
|
||
<p><span class="versionmodified added">New in version 3.3.</span></p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.pool.Pool.close">
|
||
<code class="descname">close</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.pool.Pool.close" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Prevents any more tasks from being submitted to the pool. Once all the
|
||
tasks have been completed the worker processes will exit.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.pool.Pool.terminate">
|
||
<code class="descname">terminate</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.pool.Pool.terminate" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Stops the worker processes immediately without completing outstanding
|
||
work. When the pool object is garbage collected <a class="reference internal" href="#multiprocessing.pool.Pool.terminate" title="multiprocessing.pool.Pool.terminate"><code class="xref py py-meth docutils literal notranslate"><span class="pre">terminate()</span></code></a> will be
|
||
called immediately.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.pool.Pool.join">
|
||
<code class="descname">join</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.pool.Pool.join" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Wait for the worker processes to exit. One must call <a class="reference internal" href="#multiprocessing.pool.Pool.close" title="multiprocessing.pool.Pool.close"><code class="xref py py-meth docutils literal notranslate"><span class="pre">close()</span></code></a> or
|
||
<a class="reference internal" href="#multiprocessing.pool.Pool.terminate" title="multiprocessing.pool.Pool.terminate"><code class="xref py py-meth docutils literal notranslate"><span class="pre">terminate()</span></code></a> before using <a class="reference internal" href="#multiprocessing.pool.Pool.join" title="multiprocessing.pool.Pool.join"><code class="xref py py-meth docutils literal notranslate"><span class="pre">join()</span></code></a>.</p>
|
||
</dd></dl>
|
||
|
||
<div class="versionadded">
|
||
<p><span class="versionmodified added">New in version 3.3: </span>Pool objects now support the context management protocol – see
|
||
<a class="reference internal" href="stdtypes.html#typecontextmanager"><span class="std std-ref">Context Manager Types</span></a>. <a class="reference internal" href="stdtypes.html#contextmanager.__enter__" title="contextmanager.__enter__"><code class="xref py py-meth docutils literal notranslate"><span class="pre">__enter__()</span></code></a> returns the
|
||
pool object, and <a class="reference internal" href="stdtypes.html#contextmanager.__exit__" title="contextmanager.__exit__"><code class="xref py py-meth docutils literal notranslate"><span class="pre">__exit__()</span></code></a> calls <a class="reference internal" href="#multiprocessing.pool.Pool.terminate" title="multiprocessing.pool.Pool.terminate"><code class="xref py py-meth docutils literal notranslate"><span class="pre">terminate()</span></code></a>.</p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="class">
|
||
<dt id="multiprocessing.pool.AsyncResult">
|
||
<em class="property">class </em><code class="descclassname">multiprocessing.pool.</code><code class="descname">AsyncResult</code><a class="headerlink" href="#multiprocessing.pool.AsyncResult" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>The class of the result returned by <a class="reference internal" href="#multiprocessing.pool.Pool.apply_async" title="multiprocessing.pool.Pool.apply_async"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Pool.apply_async()</span></code></a> and
|
||
<a class="reference internal" href="#multiprocessing.pool.Pool.map_async" title="multiprocessing.pool.Pool.map_async"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Pool.map_async()</span></code></a>.</p>
|
||
<dl class="method">
|
||
<dt id="multiprocessing.pool.AsyncResult.get">
|
||
<code class="descname">get</code><span class="sig-paren">(</span><span class="optional">[</span><em>timeout</em><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.pool.AsyncResult.get" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return the result when it arrives. If <em>timeout</em> is not <code class="docutils literal notranslate"><span class="pre">None</span></code> and the
|
||
result does not arrive within <em>timeout</em> seconds then
|
||
<a class="reference internal" href="#multiprocessing.TimeoutError" title="multiprocessing.TimeoutError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">multiprocessing.TimeoutError</span></code></a> is raised. If the remote call raised
|
||
an exception then that exception will be reraised by <a class="reference internal" href="#multiprocessing.pool.AsyncResult.get" title="multiprocessing.pool.AsyncResult.get"><code class="xref py py-meth docutils literal notranslate"><span class="pre">get()</span></code></a>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.pool.AsyncResult.wait">
|
||
<code class="descname">wait</code><span class="sig-paren">(</span><span class="optional">[</span><em>timeout</em><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.pool.AsyncResult.wait" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Wait until the result is available or until <em>timeout</em> seconds pass.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.pool.AsyncResult.ready">
|
||
<code class="descname">ready</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.pool.AsyncResult.ready" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return whether the call has completed.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.pool.AsyncResult.successful">
|
||
<code class="descname">successful</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.pool.AsyncResult.successful" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Return whether the call completed without raising an exception. Will
|
||
raise <a class="reference internal" href="exceptions.html#AssertionError" title="AssertionError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">AssertionError</span></code></a> if the result is not ready.</p>
|
||
</dd></dl>
|
||
|
||
</dd></dl>
|
||
|
||
<p>The following example demonstrates the use of a pool:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Pool</span>
|
||
<span class="kn">import</span> <span class="nn">time</span>
|
||
|
||
<span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="n">x</span><span class="o">*</span><span class="n">x</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="k">with</span> <span class="n">Pool</span><span class="p">(</span><span class="n">processes</span><span class="o">=</span><span class="mi">4</span><span class="p">)</span> <span class="k">as</span> <span class="n">pool</span><span class="p">:</span> <span class="c1"># start 4 worker processes</span>
|
||
<span class="n">result</span> <span class="o">=</span> <span class="n">pool</span><span class="o">.</span><span class="n">apply_async</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="p">(</span><span class="mi">10</span><span class="p">,))</span> <span class="c1"># evaluate "f(10)" asynchronously in a single process</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">result</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">timeout</span><span class="o">=</span><span class="mi">1</span><span class="p">))</span> <span class="c1"># prints "100" unless your computer is *very* slow</span>
|
||
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">pool</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)))</span> <span class="c1"># prints "[0, 1, 4,..., 81]"</span>
|
||
|
||
<span class="n">it</span> <span class="o">=</span> <span class="n">pool</span><span class="o">.</span><span class="n">imap</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">))</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="nb">next</span><span class="p">(</span><span class="n">it</span><span class="p">))</span> <span class="c1"># prints "0"</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="nb">next</span><span class="p">(</span><span class="n">it</span><span class="p">))</span> <span class="c1"># prints "1"</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">it</span><span class="o">.</span><span class="n">next</span><span class="p">(</span><span class="n">timeout</span><span class="o">=</span><span class="mi">1</span><span class="p">))</span> <span class="c1"># prints "4" unless your computer is *very* slow</span>
|
||
|
||
<span class="n">result</span> <span class="o">=</span> <span class="n">pool</span><span class="o">.</span><span class="n">apply_async</span><span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">,</span> <span class="p">(</span><span class="mi">10</span><span class="p">,))</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">result</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">timeout</span><span class="o">=</span><span class="mi">1</span><span class="p">))</span> <span class="c1"># raises multiprocessing.TimeoutError</span>
|
||
</pre></div>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="module-multiprocessing.connection">
|
||
<span id="listeners-and-clients"></span><span id="multiprocessing-listeners-clients"></span><h3>Listeners and Clients<a class="headerlink" href="#module-multiprocessing.connection" title="Permalink to this headline">¶</a></h3>
|
||
<p>Usually message passing between processes is done using queues or by using
|
||
<a class="reference internal" href="#multiprocessing.connection.Connection" title="multiprocessing.connection.Connection"><code class="xref py py-class docutils literal notranslate"><span class="pre">Connection</span></code></a> objects returned by
|
||
<a class="reference internal" href="#multiprocessing.Pipe" title="multiprocessing.Pipe"><code class="xref py py-func docutils literal notranslate"><span class="pre">Pipe()</span></code></a>.</p>
|
||
<p>However, the <a class="reference internal" href="#module-multiprocessing.connection" title="multiprocessing.connection: API for dealing with sockets."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing.connection</span></code></a> module allows some extra
|
||
flexibility. It basically gives a high level message oriented API for dealing
|
||
with sockets or Windows named pipes. It also has support for <em>digest
|
||
authentication</em> using the <a class="reference internal" href="hmac.html#module-hmac" title="hmac: Keyed-Hashing for Message Authentication (HMAC) implementation"><code class="xref py py-mod docutils literal notranslate"><span class="pre">hmac</span></code></a> module, and for polling
|
||
multiple connections at the same time.</p>
|
||
<dl class="function">
|
||
<dt id="multiprocessing.connection.deliver_challenge">
|
||
<code class="descclassname">multiprocessing.connection.</code><code class="descname">deliver_challenge</code><span class="sig-paren">(</span><em>connection</em>, <em>authkey</em><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.connection.deliver_challenge" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Send a randomly generated message to the other end of the connection and wait
|
||
for a reply.</p>
|
||
<p>If the reply matches the digest of the message using <em>authkey</em> as the key
|
||
then a welcome message is sent to the other end of the connection. Otherwise
|
||
<a class="reference internal" href="#multiprocessing.AuthenticationError" title="multiprocessing.AuthenticationError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">AuthenticationError</span></code></a> is raised.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="function">
|
||
<dt id="multiprocessing.connection.answer_challenge">
|
||
<code class="descclassname">multiprocessing.connection.</code><code class="descname">answer_challenge</code><span class="sig-paren">(</span><em>connection</em>, <em>authkey</em><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.connection.answer_challenge" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Receive a message, calculate the digest of the message using <em>authkey</em> as the
|
||
key, and then send the digest back.</p>
|
||
<p>If a welcome message is not received, then
|
||
<a class="reference internal" href="#multiprocessing.AuthenticationError" title="multiprocessing.AuthenticationError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">AuthenticationError</span></code></a> is raised.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="function">
|
||
<dt id="multiprocessing.connection.Client">
|
||
<code class="descclassname">multiprocessing.connection.</code><code class="descname">Client</code><span class="sig-paren">(</span><em>address</em><span class="optional">[</span>, <em>family</em><span class="optional">[</span>, <em>authkey</em><span class="optional">]</span><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.connection.Client" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Attempt to set up a connection to the listener which is using address
|
||
<em>address</em>, returning a <a class="reference internal" href="#multiprocessing.connection.Connection" title="multiprocessing.connection.Connection"><code class="xref py py-class docutils literal notranslate"><span class="pre">Connection</span></code></a>.</p>
|
||
<p>The type of the connection is determined by <em>family</em> argument, but this can
|
||
generally be omitted since it can usually be inferred from the format of
|
||
<em>address</em>. (See <a class="reference internal" href="#multiprocessing-address-formats"><span class="std std-ref">Address Formats</span></a>)</p>
|
||
<p>If <em>authkey</em> is given and not None, it should be a byte string and will be
|
||
used as the secret key for an HMAC-based authentication challenge. No
|
||
authentication is done if <em>authkey</em> is None.
|
||
<a class="reference internal" href="#multiprocessing.AuthenticationError" title="multiprocessing.AuthenticationError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">AuthenticationError</span></code></a> is raised if authentication fails.
|
||
See <a class="reference internal" href="#multiprocessing-auth-keys"><span class="std std-ref">Authentication keys</span></a>.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="class">
|
||
<dt id="multiprocessing.connection.Listener">
|
||
<em class="property">class </em><code class="descclassname">multiprocessing.connection.</code><code class="descname">Listener</code><span class="sig-paren">(</span><span class="optional">[</span><em>address</em><span class="optional">[</span>, <em>family</em><span class="optional">[</span>, <em>backlog</em><span class="optional">[</span>, <em>authkey</em><span class="optional">]</span><span class="optional">]</span><span class="optional">]</span><span class="optional">]</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.connection.Listener" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>A wrapper for a bound socket or Windows named pipe which is ‘listening’ for
|
||
connections.</p>
|
||
<p><em>address</em> is the address to be used by the bound socket or named pipe of the
|
||
listener object.</p>
|
||
<div class="admonition note">
|
||
<p class="admonition-title">Note</p>
|
||
<p>If an address of ‘0.0.0.0’ is used, the address will not be a connectable
|
||
end point on Windows. If you require a connectable end-point,
|
||
you should use ‘127.0.0.1’.</p>
|
||
</div>
|
||
<p><em>family</em> is the type of socket (or named pipe) to use. This can be one of
|
||
the strings <code class="docutils literal notranslate"><span class="pre">'AF_INET'</span></code> (for a TCP socket), <code class="docutils literal notranslate"><span class="pre">'AF_UNIX'</span></code> (for a Unix
|
||
domain socket) or <code class="docutils literal notranslate"><span class="pre">'AF_PIPE'</span></code> (for a Windows named pipe). Of these only
|
||
the first is guaranteed to be available. If <em>family</em> is <code class="docutils literal notranslate"><span class="pre">None</span></code> then the
|
||
family is inferred from the format of <em>address</em>. If <em>address</em> is also
|
||
<code class="docutils literal notranslate"><span class="pre">None</span></code> then a default is chosen. This default is the family which is
|
||
assumed to be the fastest available. See
|
||
<a class="reference internal" href="#multiprocessing-address-formats"><span class="std std-ref">Address Formats</span></a>. Note that if <em>family</em> is
|
||
<code class="docutils literal notranslate"><span class="pre">'AF_UNIX'</span></code> and address is <code class="docutils literal notranslate"><span class="pre">None</span></code> then the socket will be created in a
|
||
private temporary directory created using <a class="reference internal" href="tempfile.html#tempfile.mkstemp" title="tempfile.mkstemp"><code class="xref py py-func docutils literal notranslate"><span class="pre">tempfile.mkstemp()</span></code></a>.</p>
|
||
<p>If the listener object uses a socket then <em>backlog</em> (1 by default) is passed
|
||
to the <a class="reference internal" href="socket.html#socket.socket.listen" title="socket.socket.listen"><code class="xref py py-meth docutils literal notranslate"><span class="pre">listen()</span></code></a> method of the socket once it has been
|
||
bound.</p>
|
||
<p>If <em>authkey</em> is given and not None, it should be a byte string and will be
|
||
used as the secret key for an HMAC-based authentication challenge. No
|
||
authentication is done if <em>authkey</em> is None.
|
||
<a class="reference internal" href="#multiprocessing.AuthenticationError" title="multiprocessing.AuthenticationError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">AuthenticationError</span></code></a> is raised if authentication fails.
|
||
See <a class="reference internal" href="#multiprocessing-auth-keys"><span class="std std-ref">Authentication keys</span></a>.</p>
|
||
<dl class="method">
|
||
<dt id="multiprocessing.connection.Listener.accept">
|
||
<code class="descname">accept</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.connection.Listener.accept" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Accept a connection on the bound socket or named pipe of the listener
|
||
object and return a <a class="reference internal" href="#multiprocessing.connection.Connection" title="multiprocessing.connection.Connection"><code class="xref py py-class docutils literal notranslate"><span class="pre">Connection</span></code></a> object.
|
||
If authentication is attempted and fails, then
|
||
<a class="reference internal" href="#multiprocessing.AuthenticationError" title="multiprocessing.AuthenticationError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">AuthenticationError</span></code></a> is raised.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="method">
|
||
<dt id="multiprocessing.connection.Listener.close">
|
||
<code class="descname">close</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.connection.Listener.close" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Close the bound socket or named pipe of the listener object. This is
|
||
called automatically when the listener is garbage collected. However it
|
||
is advisable to call it explicitly.</p>
|
||
</dd></dl>
|
||
|
||
<p>Listener objects have the following read-only properties:</p>
|
||
<dl class="attribute">
|
||
<dt id="multiprocessing.connection.Listener.address">
|
||
<code class="descname">address</code><a class="headerlink" href="#multiprocessing.connection.Listener.address" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>The address which is being used by the Listener object.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="attribute">
|
||
<dt id="multiprocessing.connection.Listener.last_accepted">
|
||
<code class="descname">last_accepted</code><a class="headerlink" href="#multiprocessing.connection.Listener.last_accepted" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>The address from which the last accepted connection came. If this is
|
||
unavailable then it is <code class="docutils literal notranslate"><span class="pre">None</span></code>.</p>
|
||
</dd></dl>
|
||
|
||
<div class="versionadded">
|
||
<p><span class="versionmodified added">New in version 3.3: </span>Listener objects now support the context management protocol – see
|
||
<a class="reference internal" href="stdtypes.html#typecontextmanager"><span class="std std-ref">Context Manager Types</span></a>. <a class="reference internal" href="stdtypes.html#contextmanager.__enter__" title="contextmanager.__enter__"><code class="xref py py-meth docutils literal notranslate"><span class="pre">__enter__()</span></code></a> returns the
|
||
listener object, and <a class="reference internal" href="stdtypes.html#contextmanager.__exit__" title="contextmanager.__exit__"><code class="xref py py-meth docutils literal notranslate"><span class="pre">__exit__()</span></code></a> calls <a class="reference internal" href="#multiprocessing.connection.Listener.close" title="multiprocessing.connection.Listener.close"><code class="xref py py-meth docutils literal notranslate"><span class="pre">close()</span></code></a>.</p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<dl class="function">
|
||
<dt id="multiprocessing.connection.wait">
|
||
<code class="descclassname">multiprocessing.connection.</code><code class="descname">wait</code><span class="sig-paren">(</span><em>object_list</em>, <em>timeout=None</em><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.connection.wait" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Wait till an object in <em>object_list</em> is ready. Returns the list of
|
||
those objects in <em>object_list</em> which are ready. If <em>timeout</em> is a
|
||
float then the call blocks for at most that many seconds. If
|
||
<em>timeout</em> is <code class="docutils literal notranslate"><span class="pre">None</span></code> then it will block for an unlimited period.
|
||
A negative timeout is equivalent to a zero timeout.</p>
|
||
<p>For both Unix and Windows, an object can appear in <em>object_list</em> if
|
||
it is</p>
|
||
<ul class="simple">
|
||
<li><p>a readable <a class="reference internal" href="#multiprocessing.connection.Connection" title="multiprocessing.connection.Connection"><code class="xref py py-class docutils literal notranslate"><span class="pre">Connection</span></code></a> object;</p></li>
|
||
<li><p>a connected and readable <a class="reference internal" href="socket.html#socket.socket" title="socket.socket"><code class="xref py py-class docutils literal notranslate"><span class="pre">socket.socket</span></code></a> object; or</p></li>
|
||
<li><p>the <a class="reference internal" href="#multiprocessing.Process.sentinel" title="multiprocessing.Process.sentinel"><code class="xref py py-attr docutils literal notranslate"><span class="pre">sentinel</span></code></a> attribute of a
|
||
<a class="reference internal" href="#multiprocessing.Process" title="multiprocessing.Process"><code class="xref py py-class docutils literal notranslate"><span class="pre">Process</span></code></a> object.</p></li>
|
||
</ul>
|
||
<p>A connection or socket object is ready when there is data available
|
||
to be read from it, or the other end has been closed.</p>
|
||
<p><strong>Unix</strong>: <code class="docutils literal notranslate"><span class="pre">wait(object_list,</span> <span class="pre">timeout)</span></code> almost equivalent
|
||
<code class="docutils literal notranslate"><span class="pre">select.select(object_list,</span> <span class="pre">[],</span> <span class="pre">[],</span> <span class="pre">timeout)</span></code>. The difference is
|
||
that, if <a class="reference internal" href="select.html#select.select" title="select.select"><code class="xref py py-func docutils literal notranslate"><span class="pre">select.select()</span></code></a> is interrupted by a signal, it can
|
||
raise <a class="reference internal" href="exceptions.html#OSError" title="OSError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">OSError</span></code></a> with an error number of <code class="docutils literal notranslate"><span class="pre">EINTR</span></code>, whereas
|
||
<a class="reference internal" href="#multiprocessing.connection.wait" title="multiprocessing.connection.wait"><code class="xref py py-func docutils literal notranslate"><span class="pre">wait()</span></code></a> will not.</p>
|
||
<p><strong>Windows</strong>: An item in <em>object_list</em> must either be an integer
|
||
handle which is waitable (according to the definition used by the
|
||
documentation of the Win32 function <code class="docutils literal notranslate"><span class="pre">WaitForMultipleObjects()</span></code>)
|
||
or it can be an object with a <code class="xref py py-meth docutils literal notranslate"><span class="pre">fileno()</span></code> method which returns a
|
||
socket handle or pipe handle. (Note that pipe handles and socket
|
||
handles are <strong>not</strong> waitable handles.)</p>
|
||
<div class="versionadded">
|
||
<p><span class="versionmodified added">New in version 3.3.</span></p>
|
||
</div>
|
||
</dd></dl>
|
||
|
||
<p><strong>Examples</strong></p>
|
||
<p>The following server code creates a listener which uses <code class="docutils literal notranslate"><span class="pre">'secret</span> <span class="pre">password'</span></code> as
|
||
an authentication key. It then waits for a connection and sends some data to
|
||
the client:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing.connection</span> <span class="k">import</span> <span class="n">Listener</span>
|
||
<span class="kn">from</span> <span class="nn">array</span> <span class="k">import</span> <span class="n">array</span>
|
||
|
||
<span class="n">address</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'localhost'</span><span class="p">,</span> <span class="mi">6000</span><span class="p">)</span> <span class="c1"># family is deduced to be 'AF_INET'</span>
|
||
|
||
<span class="k">with</span> <span class="n">Listener</span><span class="p">(</span><span class="n">address</span><span class="p">,</span> <span class="n">authkey</span><span class="o">=</span><span class="sa">b</span><span class="s1">'secret password'</span><span class="p">)</span> <span class="k">as</span> <span class="n">listener</span><span class="p">:</span>
|
||
<span class="k">with</span> <span class="n">listener</span><span class="o">.</span><span class="n">accept</span><span class="p">()</span> <span class="k">as</span> <span class="n">conn</span><span class="p">:</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'connection accepted from'</span><span class="p">,</span> <span class="n">listener</span><span class="o">.</span><span class="n">last_accepted</span><span class="p">)</span>
|
||
|
||
<span class="n">conn</span><span class="o">.</span><span class="n">send</span><span class="p">([</span><span class="mf">2.25</span><span class="p">,</span> <span class="kc">None</span><span class="p">,</span> <span class="s1">'junk'</span><span class="p">,</span> <span class="nb">float</span><span class="p">])</span>
|
||
|
||
<span class="n">conn</span><span class="o">.</span><span class="n">send_bytes</span><span class="p">(</span><span class="sa">b</span><span class="s1">'hello'</span><span class="p">)</span>
|
||
|
||
<span class="n">conn</span><span class="o">.</span><span class="n">send_bytes</span><span class="p">(</span><span class="n">array</span><span class="p">(</span><span class="s1">'i'</span><span class="p">,</span> <span class="p">[</span><span class="mi">42</span><span class="p">,</span> <span class="mi">1729</span><span class="p">]))</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>The following code connects to the server and receives some data from the
|
||
server:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing.connection</span> <span class="k">import</span> <span class="n">Client</span>
|
||
<span class="kn">from</span> <span class="nn">array</span> <span class="k">import</span> <span class="n">array</span>
|
||
|
||
<span class="n">address</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'localhost'</span><span class="p">,</span> <span class="mi">6000</span><span class="p">)</span>
|
||
|
||
<span class="k">with</span> <span class="n">Client</span><span class="p">(</span><span class="n">address</span><span class="p">,</span> <span class="n">authkey</span><span class="o">=</span><span class="sa">b</span><span class="s1">'secret password'</span><span class="p">)</span> <span class="k">as</span> <span class="n">conn</span><span class="p">:</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">recv</span><span class="p">())</span> <span class="c1"># => [2.25, None, 'junk', float]</span>
|
||
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">recv_bytes</span><span class="p">())</span> <span class="c1"># => 'hello'</span>
|
||
|
||
<span class="n">arr</span> <span class="o">=</span> <span class="n">array</span><span class="p">(</span><span class="s1">'i'</span><span class="p">,</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">])</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">recv_bytes_into</span><span class="p">(</span><span class="n">arr</span><span class="p">))</span> <span class="c1"># => 8</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">arr</span><span class="p">)</span> <span class="c1"># => array('i', [42, 1729, 0, 0, 0])</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>The following code uses <a class="reference internal" href="#multiprocessing.connection.wait" title="multiprocessing.connection.wait"><code class="xref py py-func docutils literal notranslate"><span class="pre">wait()</span></code></a> to
|
||
wait for messages from multiple processes at once:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">time</span><span class="o">,</span> <span class="nn">random</span>
|
||
<span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Process</span><span class="p">,</span> <span class="n">Pipe</span><span class="p">,</span> <span class="n">current_process</span>
|
||
<span class="kn">from</span> <span class="nn">multiprocessing.connection</span> <span class="k">import</span> <span class="n">wait</span>
|
||
|
||
<span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="n">w</span><span class="p">):</span>
|
||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span>
|
||
<span class="n">w</span><span class="o">.</span><span class="n">send</span><span class="p">((</span><span class="n">i</span><span class="p">,</span> <span class="n">current_process</span><span class="p">()</span><span class="o">.</span><span class="n">name</span><span class="p">))</span>
|
||
<span class="n">w</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="n">readers</span> <span class="o">=</span> <span class="p">[]</span>
|
||
|
||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">4</span><span class="p">):</span>
|
||
<span class="n">r</span><span class="p">,</span> <span class="n">w</span> <span class="o">=</span> <span class="n">Pipe</span><span class="p">(</span><span class="n">duplex</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
|
||
<span class="n">readers</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">r</span><span class="p">)</span>
|
||
<span class="n">p</span> <span class="o">=</span> <span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">foo</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">w</span><span class="p">,))</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
|
||
<span class="c1"># We close the writable end of the pipe now to be sure that</span>
|
||
<span class="c1"># p is the only process which owns a handle for it. This</span>
|
||
<span class="c1"># ensures that when p closes its handle for the writable end,</span>
|
||
<span class="c1"># wait() will promptly report the readable end as being ready.</span>
|
||
<span class="n">w</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
|
||
|
||
<span class="k">while</span> <span class="n">readers</span><span class="p">:</span>
|
||
<span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">wait</span><span class="p">(</span><span class="n">readers</span><span class="p">):</span>
|
||
<span class="k">try</span><span class="p">:</span>
|
||
<span class="n">msg</span> <span class="o">=</span> <span class="n">r</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
|
||
<span class="k">except</span> <span class="ne">EOFError</span><span class="p">:</span>
|
||
<span class="n">readers</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">r</span><span class="p">)</span>
|
||
<span class="k">else</span><span class="p">:</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span>
|
||
</pre></div>
|
||
</div>
|
||
<div class="section" id="address-formats">
|
||
<span id="multiprocessing-address-formats"></span><h4>Address Formats<a class="headerlink" href="#address-formats" title="Permalink to this headline">¶</a></h4>
|
||
<ul class="simple">
|
||
<li><p>An <code class="docutils literal notranslate"><span class="pre">'AF_INET'</span></code> address is a tuple of the form <code class="docutils literal notranslate"><span class="pre">(hostname,</span> <span class="pre">port)</span></code> where
|
||
<em>hostname</em> is a string and <em>port</em> is an integer.</p></li>
|
||
<li><p>An <code class="docutils literal notranslate"><span class="pre">'AF_UNIX'</span></code> address is a string representing a filename on the
|
||
filesystem.</p></li>
|
||
<li><dl class="simple">
|
||
<dt>An <code class="docutils literal notranslate"><span class="pre">'AF_PIPE'</span></code> address is a string of the form</dt><dd><p><code class="samp docutils literal notranslate"><span class="pre">r'\.\pipe{PipeName}'</span></code>. To use <a class="reference internal" href="#multiprocessing.connection.Client" title="multiprocessing.connection.Client"><code class="xref py py-func docutils literal notranslate"><span class="pre">Client()</span></code></a> to connect to a named
|
||
pipe on a remote computer called <em>ServerName</em> one should use an address of the
|
||
form <code class="samp docutils literal notranslate"><span class="pre">r'\</span><em><span class="pre">ServerName</span></em><span class="pre">\pipe{PipeName}'</span></code> instead.</p>
|
||
</dd>
|
||
</dl>
|
||
</li>
|
||
</ul>
|
||
<p>Note that any string beginning with two backslashes is assumed by default to be
|
||
an <code class="docutils literal notranslate"><span class="pre">'AF_PIPE'</span></code> address rather than an <code class="docutils literal notranslate"><span class="pre">'AF_UNIX'</span></code> address.</p>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="authentication-keys">
|
||
<span id="multiprocessing-auth-keys"></span><h3>Authentication keys<a class="headerlink" href="#authentication-keys" title="Permalink to this headline">¶</a></h3>
|
||
<p>When one uses <a class="reference internal" href="#multiprocessing.connection.Connection.recv" title="multiprocessing.connection.Connection.recv"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Connection.recv</span></code></a>, the
|
||
data received is automatically
|
||
unpickled. Unfortunately unpickling data from an untrusted source is a security
|
||
risk. Therefore <a class="reference internal" href="#multiprocessing.connection.Listener" title="multiprocessing.connection.Listener"><code class="xref py py-class docutils literal notranslate"><span class="pre">Listener</span></code></a> and <a class="reference internal" href="#multiprocessing.connection.Client" title="multiprocessing.connection.Client"><code class="xref py py-func docutils literal notranslate"><span class="pre">Client()</span></code></a> use the <a class="reference internal" href="hmac.html#module-hmac" title="hmac: Keyed-Hashing for Message Authentication (HMAC) implementation"><code class="xref py py-mod docutils literal notranslate"><span class="pre">hmac</span></code></a> module
|
||
to provide digest authentication.</p>
|
||
<p>An authentication key is a byte string which can be thought of as a
|
||
password: once a connection is established both ends will demand proof
|
||
that the other knows the authentication key. (Demonstrating that both
|
||
ends are using the same key does <strong>not</strong> involve sending the key over
|
||
the connection.)</p>
|
||
<p>If authentication is requested but no authentication key is specified then the
|
||
return value of <code class="docutils literal notranslate"><span class="pre">current_process().authkey</span></code> is used (see
|
||
<a class="reference internal" href="#multiprocessing.Process" title="multiprocessing.Process"><code class="xref py py-class docutils literal notranslate"><span class="pre">Process</span></code></a>). This value will be automatically inherited by
|
||
any <a class="reference internal" href="#multiprocessing.Process" title="multiprocessing.Process"><code class="xref py py-class docutils literal notranslate"><span class="pre">Process</span></code></a> object that the current process creates.
|
||
This means that (by default) all processes of a multi-process program will share
|
||
a single authentication key which can be used when setting up connections
|
||
between themselves.</p>
|
||
<p>Suitable authentication keys can also be generated by using <a class="reference internal" href="os.html#os.urandom" title="os.urandom"><code class="xref py py-func docutils literal notranslate"><span class="pre">os.urandom()</span></code></a>.</p>
|
||
</div>
|
||
<div class="section" id="logging">
|
||
<h3>Logging<a class="headerlink" href="#logging" title="Permalink to this headline">¶</a></h3>
|
||
<p>Some support for logging is available. Note, however, that the <a class="reference internal" href="logging.html#module-logging" title="logging: Flexible event logging system for applications."><code class="xref py py-mod docutils literal notranslate"><span class="pre">logging</span></code></a>
|
||
package does not use process shared locks so it is possible (depending on the
|
||
handler type) for messages from different processes to get mixed up.</p>
|
||
<dl class="function">
|
||
<dt id="multiprocessing.get_logger">
|
||
<code class="descclassname">multiprocessing.</code><code class="descname">get_logger</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.get_logger" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>Returns the logger used by <a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a>. If necessary, a new one
|
||
will be created.</p>
|
||
<p>When first created the logger has level <code class="xref py py-data docutils literal notranslate"><span class="pre">logging.NOTSET</span></code> and no
|
||
default handler. Messages sent to this logger will not by default propagate
|
||
to the root logger.</p>
|
||
<p>Note that on Windows child processes will only inherit the level of the
|
||
parent process’s logger – any other customization of the logger will not be
|
||
inherited.</p>
|
||
</dd></dl>
|
||
|
||
<dl class="function">
|
||
<dt id="multiprocessing.log_to_stderr">
|
||
<code class="descclassname">multiprocessing.</code><code class="descname">log_to_stderr</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#multiprocessing.log_to_stderr" title="Permalink to this definition">¶</a></dt>
|
||
<dd><p>This function performs a call to <a class="reference internal" href="#multiprocessing.get_logger" title="multiprocessing.get_logger"><code class="xref py py-func docutils literal notranslate"><span class="pre">get_logger()</span></code></a> but in addition to
|
||
returning the logger created by get_logger, it adds a handler which sends
|
||
output to <a class="reference internal" href="sys.html#sys.stderr" title="sys.stderr"><code class="xref py py-data docutils literal notranslate"><span class="pre">sys.stderr</span></code></a> using format
|
||
<code class="docutils literal notranslate"><span class="pre">'[%(levelname)s/%(processName)s]</span> <span class="pre">%(message)s'</span></code>.</p>
|
||
</dd></dl>
|
||
|
||
<p>Below is an example session with logging turned on:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">multiprocessing</span><span class="o">,</span> <span class="nn">logging</span>
|
||
<span class="gp">>>> </span><span class="n">logger</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">log_to_stderr</span><span class="p">()</span>
|
||
<span class="gp">>>> </span><span class="n">logger</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">)</span>
|
||
<span class="gp">>>> </span><span class="n">logger</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="s1">'doomed'</span><span class="p">)</span>
|
||
<span class="go">[WARNING/MainProcess] doomed</span>
|
||
<span class="gp">>>> </span><span class="n">m</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Manager</span><span class="p">()</span>
|
||
<span class="go">[INFO/SyncManager-...] child process calling self.run()</span>
|
||
<span class="go">[INFO/SyncManager-...] created temp directory /.../pymp-...</span>
|
||
<span class="go">[INFO/SyncManager-...] manager serving at '/.../listener-...'</span>
|
||
<span class="gp">>>> </span><span class="k">del</span> <span class="n">m</span>
|
||
<span class="go">[INFO/MainProcess] sending shutdown message to manager</span>
|
||
<span class="go">[INFO/SyncManager-...] manager exiting with exitcode 0</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>For a full table of logging levels, see the <a class="reference internal" href="logging.html#module-logging" title="logging: Flexible event logging system for applications."><code class="xref py py-mod docutils literal notranslate"><span class="pre">logging</span></code></a> module.</p>
|
||
</div>
|
||
<div class="section" id="module-multiprocessing.dummy">
|
||
<span id="the-multiprocessing-dummy-module"></span><h3>The <a class="reference internal" href="#module-multiprocessing.dummy" title="multiprocessing.dummy: Dumb wrapper around threading."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing.dummy</span></code></a> module<a class="headerlink" href="#module-multiprocessing.dummy" title="Permalink to this headline">¶</a></h3>
|
||
<p><a class="reference internal" href="#module-multiprocessing.dummy" title="multiprocessing.dummy: Dumb wrapper around threading."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing.dummy</span></code></a> replicates the API of <a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a> but is
|
||
no more than a wrapper around the <a class="reference internal" href="threading.html#module-threading" title="threading: Thread-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">threading</span></code></a> module.</p>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="programming-guidelines">
|
||
<span id="multiprocessing-programming"></span><h2>Programming guidelines<a class="headerlink" href="#programming-guidelines" title="Permalink to this headline">¶</a></h2>
|
||
<p>There are certain guidelines and idioms which should be adhered to when using
|
||
<a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a>.</p>
|
||
<div class="section" id="all-start-methods">
|
||
<h3>All start methods<a class="headerlink" href="#all-start-methods" title="Permalink to this headline">¶</a></h3>
|
||
<p>The following applies to all start methods.</p>
|
||
<p>Avoid shared state</p>
|
||
<blockquote>
|
||
<div><p>As far as possible one should try to avoid shifting large amounts of data
|
||
between processes.</p>
|
||
<p>It is probably best to stick to using queues or pipes for communication
|
||
between processes rather than using the lower level synchronization
|
||
primitives.</p>
|
||
</div></blockquote>
|
||
<p>Picklability</p>
|
||
<blockquote>
|
||
<div><p>Ensure that the arguments to the methods of proxies are picklable.</p>
|
||
</div></blockquote>
|
||
<p>Thread safety of proxies</p>
|
||
<blockquote>
|
||
<div><p>Do not use a proxy object from more than one thread unless you protect it
|
||
with a lock.</p>
|
||
<p>(There is never a problem with different processes using the <em>same</em> proxy.)</p>
|
||
</div></blockquote>
|
||
<p>Joining zombie processes</p>
|
||
<blockquote>
|
||
<div><p>On Unix when a process finishes but has not been joined it becomes a zombie.
|
||
There should never be very many because each time a new process starts (or
|
||
<a class="reference internal" href="#multiprocessing.active_children" title="multiprocessing.active_children"><code class="xref py py-func docutils literal notranslate"><span class="pre">active_children()</span></code></a> is called) all completed processes
|
||
which have not yet been joined will be joined. Also calling a finished
|
||
process’s <a class="reference internal" href="#multiprocessing.Process.is_alive" title="multiprocessing.Process.is_alive"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Process.is_alive</span></code></a> will
|
||
join the process. Even so it is probably good
|
||
practice to explicitly join all the processes that you start.</p>
|
||
</div></blockquote>
|
||
<p>Better to inherit than pickle/unpickle</p>
|
||
<blockquote>
|
||
<div><p>When using the <em>spawn</em> or <em>forkserver</em> start methods many types
|
||
from <a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a> need to be picklable so that child
|
||
processes can use them. However, one should generally avoid
|
||
sending shared objects to other processes using pipes or queues.
|
||
Instead you should arrange the program so that a process which
|
||
needs access to a shared resource created elsewhere can inherit it
|
||
from an ancestor process.</p>
|
||
</div></blockquote>
|
||
<p>Avoid terminating processes</p>
|
||
<blockquote>
|
||
<div><p>Using the <a class="reference internal" href="#multiprocessing.Process.terminate" title="multiprocessing.Process.terminate"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Process.terminate</span></code></a>
|
||
method to stop a process is liable to
|
||
cause any shared resources (such as locks, semaphores, pipes and queues)
|
||
currently being used by the process to become broken or unavailable to other
|
||
processes.</p>
|
||
<p>Therefore it is probably best to only consider using
|
||
<a class="reference internal" href="#multiprocessing.Process.terminate" title="multiprocessing.Process.terminate"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Process.terminate</span></code></a> on processes
|
||
which never use any shared resources.</p>
|
||
</div></blockquote>
|
||
<p>Joining processes that use queues</p>
|
||
<blockquote>
|
||
<div><p>Bear in mind that a process that has put items in a queue will wait before
|
||
terminating until all the buffered items are fed by the “feeder” thread to
|
||
the underlying pipe. (The child process can call the
|
||
<a class="reference internal" href="#multiprocessing.Queue.cancel_join_thread" title="multiprocessing.Queue.cancel_join_thread"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Queue.cancel_join_thread</span></code></a>
|
||
method of the queue to avoid this behaviour.)</p>
|
||
<p>This means that whenever you use a queue you need to make sure that all
|
||
items which have been put on the queue will eventually be removed before the
|
||
process is joined. Otherwise you cannot be sure that processes which have
|
||
put items on the queue will terminate. Remember also that non-daemonic
|
||
processes will be joined automatically.</p>
|
||
<p>An example which will deadlock is the following:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Process</span><span class="p">,</span> <span class="n">Queue</span>
|
||
|
||
<span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">q</span><span class="p">):</span>
|
||
<span class="n">q</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'X'</span> <span class="o">*</span> <span class="mi">1000000</span><span class="p">)</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="n">queue</span> <span class="o">=</span> <span class="n">Queue</span><span class="p">()</span>
|
||
<span class="n">p</span> <span class="o">=</span> <span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">f</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">queue</span><span class="p">,))</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span> <span class="c1"># this deadlocks</span>
|
||
<span class="n">obj</span> <span class="o">=</span> <span class="n">queue</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>A fix here would be to swap the last two lines (or simply remove the
|
||
<code class="docutils literal notranslate"><span class="pre">p.join()</span></code> line).</p>
|
||
</div></blockquote>
|
||
<p>Explicitly pass resources to child processes</p>
|
||
<blockquote>
|
||
<div><p>On Unix using the <em>fork</em> start method, a child process can make
|
||
use of a shared resource created in a parent process using a
|
||
global resource. However, it is better to pass the object as an
|
||
argument to the constructor for the child process.</p>
|
||
<p>Apart from making the code (potentially) compatible with Windows
|
||
and the other start methods this also ensures that as long as the
|
||
child process is still alive the object will not be garbage
|
||
collected in the parent process. This might be important if some
|
||
resource is freed when the object is garbage collected in the
|
||
parent process.</p>
|
||
<p>So for instance</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Process</span><span class="p">,</span> <span class="n">Lock</span>
|
||
|
||
<span class="k">def</span> <span class="nf">f</span><span class="p">():</span>
|
||
<span class="o">...</span> <span class="n">do</span> <span class="n">something</span> <span class="n">using</span> <span class="s2">"lock"</span> <span class="o">...</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="n">lock</span> <span class="o">=</span> <span class="n">Lock</span><span class="p">()</span>
|
||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span>
|
||
<span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">f</span><span class="p">)</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>should be rewritten as</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Process</span><span class="p">,</span> <span class="n">Lock</span>
|
||
|
||
<span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">l</span><span class="p">):</span>
|
||
<span class="o">...</span> <span class="n">do</span> <span class="n">something</span> <span class="n">using</span> <span class="s2">"l"</span> <span class="o">...</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="n">lock</span> <span class="o">=</span> <span class="n">Lock</span><span class="p">()</span>
|
||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span>
|
||
<span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">f</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">lock</span><span class="p">,))</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
|
||
</pre></div>
|
||
</div>
|
||
</div></blockquote>
|
||
<p>Beware of replacing <a class="reference internal" href="sys.html#sys.stdin" title="sys.stdin"><code class="xref py py-data docutils literal notranslate"><span class="pre">sys.stdin</span></code></a> with a “file like object”</p>
|
||
<blockquote>
|
||
<div><p><a class="reference internal" href="#module-multiprocessing" title="multiprocessing: Process-based parallelism."><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code></a> originally unconditionally called:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="n">os</span><span class="o">.</span><span class="n">close</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">stdin</span><span class="o">.</span><span class="n">fileno</span><span class="p">())</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>in the <code class="xref py py-meth docutils literal notranslate"><span class="pre">multiprocessing.Process._bootstrap()</span></code> method — this resulted
|
||
in issues with processes-in-processes. This has been changed to:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="n">sys</span><span class="o">.</span><span class="n">stdin</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
|
||
<span class="n">sys</span><span class="o">.</span><span class="n">stdin</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">devnull</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">O_RDONLY</span><span class="p">),</span> <span class="n">closefd</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>Which solves the fundamental issue of processes colliding with each other
|
||
resulting in a bad file descriptor error, but introduces a potential danger
|
||
to applications which replace <a class="reference internal" href="sys.html#sys.stdin" title="sys.stdin"><code class="xref py py-func docutils literal notranslate"><span class="pre">sys.stdin()</span></code></a> with a “file-like object”
|
||
with output buffering. This danger is that if multiple processes call
|
||
<a class="reference internal" href="io.html#io.IOBase.close" title="io.IOBase.close"><code class="xref py py-meth docutils literal notranslate"><span class="pre">close()</span></code></a> on this file-like object, it could result in the same
|
||
data being flushed to the object multiple times, resulting in corruption.</p>
|
||
<p>If you write a file-like object and implement your own caching, you can
|
||
make it fork-safe by storing the pid whenever you append to the cache,
|
||
and discarding the cache when the pid changes. For example:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="nd">@property</span>
|
||
<span class="k">def</span> <span class="nf">cache</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="n">pid</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getpid</span><span class="p">()</span>
|
||
<span class="k">if</span> <span class="n">pid</span> <span class="o">!=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_pid</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">_pid</span> <span class="o">=</span> <span class="n">pid</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">_cache</span> <span class="o">=</span> <span class="p">[]</span>
|
||
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_cache</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>For more information, see <a class="reference external" href="https://bugs.python.org/issue5155">bpo-5155</a>, <a class="reference external" href="https://bugs.python.org/issue5313">bpo-5313</a> and <a class="reference external" href="https://bugs.python.org/issue5331">bpo-5331</a></p>
|
||
</div></blockquote>
|
||
</div>
|
||
<div class="section" id="the-spawn-and-forkserver-start-methods">
|
||
<h3>The <em>spawn</em> and <em>forkserver</em> start methods<a class="headerlink" href="#the-spawn-and-forkserver-start-methods" title="Permalink to this headline">¶</a></h3>
|
||
<p>There are a few extra restriction which don’t apply to the <em>fork</em>
|
||
start method.</p>
|
||
<p>More picklability</p>
|
||
<blockquote>
|
||
<div><p>Ensure that all arguments to <code class="xref py py-meth docutils literal notranslate"><span class="pre">Process.__init__()</span></code> are picklable.
|
||
Also, if you subclass <a class="reference internal" href="#multiprocessing.Process" title="multiprocessing.Process"><code class="xref py py-class docutils literal notranslate"><span class="pre">Process</span></code></a> then make sure that
|
||
instances will be picklable when the <a class="reference internal" href="#multiprocessing.Process.start" title="multiprocessing.Process.start"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Process.start</span></code></a> method is called.</p>
|
||
</div></blockquote>
|
||
<p>Global variables</p>
|
||
<blockquote>
|
||
<div><p>Bear in mind that if code run in a child process tries to access a global
|
||
variable, then the value it sees (if any) may not be the same as the value
|
||
in the parent process at the time that <a class="reference internal" href="#multiprocessing.Process.start" title="multiprocessing.Process.start"><code class="xref py py-meth docutils literal notranslate"><span class="pre">Process.start</span></code></a> was called.</p>
|
||
<p>However, global variables which are just module level constants cause no
|
||
problems.</p>
|
||
</div></blockquote>
|
||
<p>Safe importing of main module</p>
|
||
<blockquote>
|
||
<div><p>Make sure that the main module can be safely imported by a new Python
|
||
interpreter without causing unintended side effects (such a starting a new
|
||
process).</p>
|
||
<p>For example, using the <em>spawn</em> or <em>forkserver</em> start method
|
||
running the following module would fail with a
|
||
<a class="reference internal" href="exceptions.html#RuntimeError" title="RuntimeError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">RuntimeError</span></code></a>:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Process</span>
|
||
|
||
<span class="k">def</span> <span class="nf">foo</span><span class="p">():</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">)</span>
|
||
|
||
<span class="n">p</span> <span class="o">=</span> <span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">foo</span><span class="p">)</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>Instead one should protect the “entry point” of the program by using <code class="docutils literal notranslate"><span class="pre">if</span>
|
||
<span class="pre">__name__</span> <span class="pre">==</span> <span class="pre">'__main__':</span></code> as follows:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Process</span><span class="p">,</span> <span class="n">freeze_support</span><span class="p">,</span> <span class="n">set_start_method</span>
|
||
|
||
<span class="k">def</span> <span class="nf">foo</span><span class="p">():</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'hello'</span><span class="p">)</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="n">freeze_support</span><span class="p">()</span>
|
||
<span class="n">set_start_method</span><span class="p">(</span><span class="s1">'spawn'</span><span class="p">)</span>
|
||
<span class="n">p</span> <span class="o">=</span> <span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">foo</span><span class="p">)</span>
|
||
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>(The <code class="docutils literal notranslate"><span class="pre">freeze_support()</span></code> line can be omitted if the program will be run
|
||
normally instead of frozen.)</p>
|
||
<p>This allows the newly spawned Python interpreter to safely import the module
|
||
and then run the module’s <code class="docutils literal notranslate"><span class="pre">foo()</span></code> function.</p>
|
||
<p>Similar restrictions apply if a pool or manager is created in the main
|
||
module.</p>
|
||
</div></blockquote>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="examples">
|
||
<span id="multiprocessing-examples"></span><h2>Examples<a class="headerlink" href="#examples" title="Permalink to this headline">¶</a></h2>
|
||
<p>Demonstration of how to create and use customized managers and proxies:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">freeze_support</span>
|
||
<span class="kn">from</span> <span class="nn">multiprocessing.managers</span> <span class="k">import</span> <span class="n">BaseManager</span><span class="p">,</span> <span class="n">BaseProxy</span>
|
||
<span class="kn">import</span> <span class="nn">operator</span>
|
||
|
||
<span class="c1">##</span>
|
||
|
||
<span class="k">class</span> <span class="nc">Foo</span><span class="p">:</span>
|
||
<span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'you called Foo.f()'</span><span class="p">)</span>
|
||
<span class="k">def</span> <span class="nf">g</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'you called Foo.g()'</span><span class="p">)</span>
|
||
<span class="k">def</span> <span class="nf">_h</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'you called Foo._h()'</span><span class="p">)</span>
|
||
|
||
<span class="c1"># A simple generator function</span>
|
||
<span class="k">def</span> <span class="nf">baz</span><span class="p">():</span>
|
||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span>
|
||
<span class="k">yield</span> <span class="n">i</span><span class="o">*</span><span class="n">i</span>
|
||
|
||
<span class="c1"># Proxy type for generator objects</span>
|
||
<span class="k">class</span> <span class="nc">GeneratorProxy</span><span class="p">(</span><span class="n">BaseProxy</span><span class="p">):</span>
|
||
<span class="n">_exposed_</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'__next__'</span><span class="p">]</span>
|
||
<span class="k">def</span> <span class="nf">__iter__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="bp">self</span>
|
||
<span class="k">def</span> <span class="nf">__next__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_callmethod</span><span class="p">(</span><span class="s1">'__next__'</span><span class="p">)</span>
|
||
|
||
<span class="c1"># Function to return the operator module</span>
|
||
<span class="k">def</span> <span class="nf">get_operator_module</span><span class="p">():</span>
|
||
<span class="k">return</span> <span class="n">operator</span>
|
||
|
||
<span class="c1">##</span>
|
||
|
||
<span class="k">class</span> <span class="nc">MyManager</span><span class="p">(</span><span class="n">BaseManager</span><span class="p">):</span>
|
||
<span class="k">pass</span>
|
||
|
||
<span class="c1"># register the Foo class; make `f()` and `g()` accessible via proxy</span>
|
||
<span class="n">MyManager</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="s1">'Foo1'</span><span class="p">,</span> <span class="n">Foo</span><span class="p">)</span>
|
||
|
||
<span class="c1"># register the Foo class; make `g()` and `_h()` accessible via proxy</span>
|
||
<span class="n">MyManager</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="s1">'Foo2'</span><span class="p">,</span> <span class="n">Foo</span><span class="p">,</span> <span class="n">exposed</span><span class="o">=</span><span class="p">(</span><span class="s1">'g'</span><span class="p">,</span> <span class="s1">'_h'</span><span class="p">))</span>
|
||
|
||
<span class="c1"># register the generator function baz; use `GeneratorProxy` to make proxies</span>
|
||
<span class="n">MyManager</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="s1">'baz'</span><span class="p">,</span> <span class="n">baz</span><span class="p">,</span> <span class="n">proxytype</span><span class="o">=</span><span class="n">GeneratorProxy</span><span class="p">)</span>
|
||
|
||
<span class="c1"># register get_operator_module(); make public functions accessible via proxy</span>
|
||
<span class="n">MyManager</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="s1">'operator'</span><span class="p">,</span> <span class="n">get_operator_module</span><span class="p">)</span>
|
||
|
||
<span class="c1">##</span>
|
||
|
||
<span class="k">def</span> <span class="nf">test</span><span class="p">():</span>
|
||
<span class="n">manager</span> <span class="o">=</span> <span class="n">MyManager</span><span class="p">()</span>
|
||
<span class="n">manager</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
|
||
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'-'</span> <span class="o">*</span> <span class="mi">20</span><span class="p">)</span>
|
||
|
||
<span class="n">f1</span> <span class="o">=</span> <span class="n">manager</span><span class="o">.</span><span class="n">Foo1</span><span class="p">()</span>
|
||
<span class="n">f1</span><span class="o">.</span><span class="n">f</span><span class="p">()</span>
|
||
<span class="n">f1</span><span class="o">.</span><span class="n">g</span><span class="p">()</span>
|
||
<span class="k">assert</span> <span class="ow">not</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">f1</span><span class="p">,</span> <span class="s1">'_h'</span><span class="p">)</span>
|
||
<span class="k">assert</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">f1</span><span class="o">.</span><span class="n">_exposed_</span><span class="p">)</span> <span class="o">==</span> <span class="nb">sorted</span><span class="p">([</span><span class="s1">'f'</span><span class="p">,</span> <span class="s1">'g'</span><span class="p">])</span>
|
||
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'-'</span> <span class="o">*</span> <span class="mi">20</span><span class="p">)</span>
|
||
|
||
<span class="n">f2</span> <span class="o">=</span> <span class="n">manager</span><span class="o">.</span><span class="n">Foo2</span><span class="p">()</span>
|
||
<span class="n">f2</span><span class="o">.</span><span class="n">g</span><span class="p">()</span>
|
||
<span class="n">f2</span><span class="o">.</span><span class="n">_h</span><span class="p">()</span>
|
||
<span class="k">assert</span> <span class="ow">not</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">f2</span><span class="p">,</span> <span class="s1">'f'</span><span class="p">)</span>
|
||
<span class="k">assert</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">f2</span><span class="o">.</span><span class="n">_exposed_</span><span class="p">)</span> <span class="o">==</span> <span class="nb">sorted</span><span class="p">([</span><span class="s1">'g'</span><span class="p">,</span> <span class="s1">'_h'</span><span class="p">])</span>
|
||
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'-'</span> <span class="o">*</span> <span class="mi">20</span><span class="p">)</span>
|
||
|
||
<span class="n">it</span> <span class="o">=</span> <span class="n">manager</span><span class="o">.</span><span class="n">baz</span><span class="p">()</span>
|
||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">it</span><span class="p">:</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'<</span><span class="si">%d</span><span class="s1">>'</span> <span class="o">%</span> <span class="n">i</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">' '</span><span class="p">)</span>
|
||
<span class="nb">print</span><span class="p">()</span>
|
||
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'-'</span> <span class="o">*</span> <span class="mi">20</span><span class="p">)</span>
|
||
|
||
<span class="n">op</span> <span class="o">=</span> <span class="n">manager</span><span class="o">.</span><span class="n">operator</span><span class="p">()</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'op.add(23, 45) ='</span><span class="p">,</span> <span class="n">op</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="mi">23</span><span class="p">,</span> <span class="mi">45</span><span class="p">))</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'op.pow(2, 94) ='</span><span class="p">,</span> <span class="n">op</span><span class="o">.</span><span class="n">pow</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">94</span><span class="p">))</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'op._exposed_ ='</span><span class="p">,</span> <span class="n">op</span><span class="o">.</span><span class="n">_exposed_</span><span class="p">)</span>
|
||
|
||
<span class="c1">##</span>
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="n">freeze_support</span><span class="p">()</span>
|
||
<span class="n">test</span><span class="p">()</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>Using <a class="reference internal" href="#multiprocessing.pool.Pool" title="multiprocessing.pool.Pool"><code class="xref py py-class docutils literal notranslate"><span class="pre">Pool</span></code></a>:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">multiprocessing</span>
|
||
<span class="kn">import</span> <span class="nn">time</span>
|
||
<span class="kn">import</span> <span class="nn">random</span>
|
||
<span class="kn">import</span> <span class="nn">sys</span>
|
||
|
||
<span class="c1">#</span>
|
||
<span class="c1"># Functions used by test code</span>
|
||
<span class="c1">#</span>
|
||
|
||
<span class="k">def</span> <span class="nf">calculate</span><span class="p">(</span><span class="n">func</span><span class="p">,</span> <span class="n">args</span><span class="p">):</span>
|
||
<span class="n">result</span> <span class="o">=</span> <span class="n">func</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">)</span>
|
||
<span class="k">return</span> <span class="s1">'</span><span class="si">%s</span><span class="s1"> says that </span><span class="si">%s%s</span><span class="s1"> = </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span>
|
||
<span class="n">multiprocessing</span><span class="o">.</span><span class="n">current_process</span><span class="p">()</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
|
||
<span class="n">func</span><span class="o">.</span><span class="vm">__name__</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="n">result</span>
|
||
<span class="p">)</span>
|
||
|
||
<span class="k">def</span> <span class="nf">calculatestar</span><span class="p">(</span><span class="n">args</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="n">calculate</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">)</span>
|
||
|
||
<span class="k">def</span> <span class="nf">mul</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
|
||
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.5</span> <span class="o">*</span> <span class="n">random</span><span class="o">.</span><span class="n">random</span><span class="p">())</span>
|
||
<span class="k">return</span> <span class="n">a</span> <span class="o">*</span> <span class="n">b</span>
|
||
|
||
<span class="k">def</span> <span class="nf">plus</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
|
||
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.5</span> <span class="o">*</span> <span class="n">random</span><span class="o">.</span><span class="n">random</span><span class="p">())</span>
|
||
<span class="k">return</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span>
|
||
|
||
<span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="mf">1.0</span> <span class="o">/</span> <span class="p">(</span><span class="n">x</span> <span class="o">-</span> <span class="mf">5.0</span><span class="p">)</span>
|
||
|
||
<span class="k">def</span> <span class="nf">pow3</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="n">x</span> <span class="o">**</span> <span class="mi">3</span>
|
||
|
||
<span class="k">def</span> <span class="nf">noop</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
|
||
<span class="k">pass</span>
|
||
|
||
<span class="c1">#</span>
|
||
<span class="c1"># Test code</span>
|
||
<span class="c1">#</span>
|
||
|
||
<span class="k">def</span> <span class="nf">test</span><span class="p">():</span>
|
||
<span class="n">PROCESSES</span> <span class="o">=</span> <span class="mi">4</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'Creating pool with </span><span class="si">%d</span><span class="s1"> processes</span><span class="se">\n</span><span class="s1">'</span> <span class="o">%</span> <span class="n">PROCESSES</span><span class="p">)</span>
|
||
|
||
<span class="k">with</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Pool</span><span class="p">(</span><span class="n">PROCESSES</span><span class="p">)</span> <span class="k">as</span> <span class="n">pool</span><span class="p">:</span>
|
||
<span class="c1">#</span>
|
||
<span class="c1"># Tests</span>
|
||
<span class="c1">#</span>
|
||
|
||
<span class="n">TASKS</span> <span class="o">=</span> <span class="p">[(</span><span class="n">mul</span><span class="p">,</span> <span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="mi">7</span><span class="p">))</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)]</span> <span class="o">+</span> \
|
||
<span class="p">[(</span><span class="n">plus</span><span class="p">,</span> <span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="mi">8</span><span class="p">))</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)]</span>
|
||
|
||
<span class="n">results</span> <span class="o">=</span> <span class="p">[</span><span class="n">pool</span><span class="o">.</span><span class="n">apply_async</span><span class="p">(</span><span class="n">calculate</span><span class="p">,</span> <span class="n">t</span><span class="p">)</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">TASKS</span><span class="p">]</span>
|
||
<span class="n">imap_it</span> <span class="o">=</span> <span class="n">pool</span><span class="o">.</span><span class="n">imap</span><span class="p">(</span><span class="n">calculatestar</span><span class="p">,</span> <span class="n">TASKS</span><span class="p">)</span>
|
||
<span class="n">imap_unordered_it</span> <span class="o">=</span> <span class="n">pool</span><span class="o">.</span><span class="n">imap_unordered</span><span class="p">(</span><span class="n">calculatestar</span><span class="p">,</span> <span class="n">TASKS</span><span class="p">)</span>
|
||
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'Ordered results using pool.apply_async():'</span><span class="p">)</span>
|
||
<span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">results</span><span class="p">:</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="se">\t</span><span class="s1">'</span><span class="p">,</span> <span class="n">r</span><span class="o">.</span><span class="n">get</span><span class="p">())</span>
|
||
<span class="nb">print</span><span class="p">()</span>
|
||
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'Ordered results using pool.imap():'</span><span class="p">)</span>
|
||
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">imap_it</span><span class="p">:</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="se">\t</span><span class="s1">'</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span>
|
||
<span class="nb">print</span><span class="p">()</span>
|
||
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'Unordered results using pool.imap_unordered():'</span><span class="p">)</span>
|
||
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">imap_unordered_it</span><span class="p">:</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="se">\t</span><span class="s1">'</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span>
|
||
<span class="nb">print</span><span class="p">()</span>
|
||
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'Ordered results using pool.map() --- will block till complete:'</span><span class="p">)</span>
|
||
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">pool</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="n">calculatestar</span><span class="p">,</span> <span class="n">TASKS</span><span class="p">):</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="se">\t</span><span class="s1">'</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span>
|
||
<span class="nb">print</span><span class="p">()</span>
|
||
|
||
<span class="c1">#</span>
|
||
<span class="c1"># Test error handling</span>
|
||
<span class="c1">#</span>
|
||
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'Testing error handling:'</span><span class="p">)</span>
|
||
|
||
<span class="k">try</span><span class="p">:</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">pool</span><span class="o">.</span><span class="n">apply</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="p">(</span><span class="mi">5</span><span class="p">,)))</span>
|
||
<span class="k">except</span> <span class="ne">ZeroDivisionError</span><span class="p">:</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="se">\t</span><span class="s1">Got ZeroDivisionError as expected from pool.apply()'</span><span class="p">)</span>
|
||
<span class="k">else</span><span class="p">:</span>
|
||
<span class="k">raise</span> <span class="ne">AssertionError</span><span class="p">(</span><span class="s1">'expected ZeroDivisionError'</span><span class="p">)</span>
|
||
|
||
<span class="k">try</span><span class="p">:</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="n">pool</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="nb">list</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">))))</span>
|
||
<span class="k">except</span> <span class="ne">ZeroDivisionError</span><span class="p">:</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="se">\t</span><span class="s1">Got ZeroDivisionError as expected from pool.map()'</span><span class="p">)</span>
|
||
<span class="k">else</span><span class="p">:</span>
|
||
<span class="k">raise</span> <span class="ne">AssertionError</span><span class="p">(</span><span class="s1">'expected ZeroDivisionError'</span><span class="p">)</span>
|
||
|
||
<span class="k">try</span><span class="p">:</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="n">pool</span><span class="o">.</span><span class="n">imap</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="nb">list</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)))))</span>
|
||
<span class="k">except</span> <span class="ne">ZeroDivisionError</span><span class="p">:</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="se">\t</span><span class="s1">Got ZeroDivisionError as expected from list(pool.imap())'</span><span class="p">)</span>
|
||
<span class="k">else</span><span class="p">:</span>
|
||
<span class="k">raise</span> <span class="ne">AssertionError</span><span class="p">(</span><span class="s1">'expected ZeroDivisionError'</span><span class="p">)</span>
|
||
|
||
<span class="n">it</span> <span class="o">=</span> <span class="n">pool</span><span class="o">.</span><span class="n">imap</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="nb">list</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)))</span>
|
||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span>
|
||
<span class="k">try</span><span class="p">:</span>
|
||
<span class="n">x</span> <span class="o">=</span> <span class="nb">next</span><span class="p">(</span><span class="n">it</span><span class="p">)</span>
|
||
<span class="k">except</span> <span class="ne">ZeroDivisionError</span><span class="p">:</span>
|
||
<span class="k">if</span> <span class="n">i</span> <span class="o">==</span> <span class="mi">5</span><span class="p">:</span>
|
||
<span class="k">pass</span>
|
||
<span class="k">except</span> <span class="ne">StopIteration</span><span class="p">:</span>
|
||
<span class="k">break</span>
|
||
<span class="k">else</span><span class="p">:</span>
|
||
<span class="k">if</span> <span class="n">i</span> <span class="o">==</span> <span class="mi">5</span><span class="p">:</span>
|
||
<span class="k">raise</span> <span class="ne">AssertionError</span><span class="p">(</span><span class="s1">'expected ZeroDivisionError'</span><span class="p">)</span>
|
||
|
||
<span class="k">assert</span> <span class="n">i</span> <span class="o">==</span> <span class="mi">9</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="se">\t</span><span class="s1">Got ZeroDivisionError as expected from IMapIterator.next()'</span><span class="p">)</span>
|
||
<span class="nb">print</span><span class="p">()</span>
|
||
|
||
<span class="c1">#</span>
|
||
<span class="c1"># Testing timeouts</span>
|
||
<span class="c1">#</span>
|
||
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'Testing ApplyResult.get() with timeout:'</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">' '</span><span class="p">)</span>
|
||
<span class="n">res</span> <span class="o">=</span> <span class="n">pool</span><span class="o">.</span><span class="n">apply_async</span><span class="p">(</span><span class="n">calculate</span><span class="p">,</span> <span class="n">TASKS</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
|
||
<span class="k">while</span> <span class="mi">1</span><span class="p">:</span>
|
||
<span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
|
||
<span class="k">try</span><span class="p">:</span>
|
||
<span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'</span><span class="se">\n\t</span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">res</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="mf">0.02</span><span class="p">))</span>
|
||
<span class="k">break</span>
|
||
<span class="k">except</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">TimeoutError</span><span class="p">:</span>
|
||
<span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'.'</span><span class="p">)</span>
|
||
<span class="nb">print</span><span class="p">()</span>
|
||
<span class="nb">print</span><span class="p">()</span>
|
||
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'Testing IMapIterator.next() with timeout:'</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s1">' '</span><span class="p">)</span>
|
||
<span class="n">it</span> <span class="o">=</span> <span class="n">pool</span><span class="o">.</span><span class="n">imap</span><span class="p">(</span><span class="n">calculatestar</span><span class="p">,</span> <span class="n">TASKS</span><span class="p">)</span>
|
||
<span class="k">while</span> <span class="mi">1</span><span class="p">:</span>
|
||
<span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
|
||
<span class="k">try</span><span class="p">:</span>
|
||
<span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'</span><span class="se">\n\t</span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">it</span><span class="o">.</span><span class="n">next</span><span class="p">(</span><span class="mf">0.02</span><span class="p">))</span>
|
||
<span class="k">except</span> <span class="ne">StopIteration</span><span class="p">:</span>
|
||
<span class="k">break</span>
|
||
<span class="k">except</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">TimeoutError</span><span class="p">:</span>
|
||
<span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'.'</span><span class="p">)</span>
|
||
<span class="nb">print</span><span class="p">()</span>
|
||
<span class="nb">print</span><span class="p">()</span>
|
||
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="n">multiprocessing</span><span class="o">.</span><span class="n">freeze_support</span><span class="p">()</span>
|
||
<span class="n">test</span><span class="p">()</span>
|
||
</pre></div>
|
||
</div>
|
||
<p>An example showing how to use queues to feed tasks to a collection of worker
|
||
processes and collect the results:</p>
|
||
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">time</span>
|
||
<span class="kn">import</span> <span class="nn">random</span>
|
||
|
||
<span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Process</span><span class="p">,</span> <span class="n">Queue</span><span class="p">,</span> <span class="n">current_process</span><span class="p">,</span> <span class="n">freeze_support</span>
|
||
|
||
<span class="c1">#</span>
|
||
<span class="c1"># Function run by worker processes</span>
|
||
<span class="c1">#</span>
|
||
|
||
<span class="k">def</span> <span class="nf">worker</span><span class="p">(</span><span class="nb">input</span><span class="p">,</span> <span class="n">output</span><span class="p">):</span>
|
||
<span class="k">for</span> <span class="n">func</span><span class="p">,</span> <span class="n">args</span> <span class="ow">in</span> <span class="nb">iter</span><span class="p">(</span><span class="nb">input</span><span class="o">.</span><span class="n">get</span><span class="p">,</span> <span class="s1">'STOP'</span><span class="p">):</span>
|
||
<span class="n">result</span> <span class="o">=</span> <span class="n">calculate</span><span class="p">(</span><span class="n">func</span><span class="p">,</span> <span class="n">args</span><span class="p">)</span>
|
||
<span class="n">output</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
|
||
|
||
<span class="c1">#</span>
|
||
<span class="c1"># Function used to calculate result</span>
|
||
<span class="c1">#</span>
|
||
|
||
<span class="k">def</span> <span class="nf">calculate</span><span class="p">(</span><span class="n">func</span><span class="p">,</span> <span class="n">args</span><span class="p">):</span>
|
||
<span class="n">result</span> <span class="o">=</span> <span class="n">func</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">)</span>
|
||
<span class="k">return</span> <span class="s1">'</span><span class="si">%s</span><span class="s1"> says that </span><span class="si">%s%s</span><span class="s1"> = </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> \
|
||
<span class="p">(</span><span class="n">current_process</span><span class="p">()</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="n">func</span><span class="o">.</span><span class="vm">__name__</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
|
||
|
||
<span class="c1">#</span>
|
||
<span class="c1"># Functions referenced by tasks</span>
|
||
<span class="c1">#</span>
|
||
|
||
<span class="k">def</span> <span class="nf">mul</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
|
||
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.5</span><span class="o">*</span><span class="n">random</span><span class="o">.</span><span class="n">random</span><span class="p">())</span>
|
||
<span class="k">return</span> <span class="n">a</span> <span class="o">*</span> <span class="n">b</span>
|
||
|
||
<span class="k">def</span> <span class="nf">plus</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
|
||
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.5</span><span class="o">*</span><span class="n">random</span><span class="o">.</span><span class="n">random</span><span class="p">())</span>
|
||
<span class="k">return</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span>
|
||
|
||
<span class="c1">#</span>
|
||
<span class="c1">#</span>
|
||
<span class="c1">#</span>
|
||
|
||
<span class="k">def</span> <span class="nf">test</span><span class="p">():</span>
|
||
<span class="n">NUMBER_OF_PROCESSES</span> <span class="o">=</span> <span class="mi">4</span>
|
||
<span class="n">TASKS1</span> <span class="o">=</span> <span class="p">[(</span><span class="n">mul</span><span class="p">,</span> <span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="mi">7</span><span class="p">))</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">20</span><span class="p">)]</span>
|
||
<span class="n">TASKS2</span> <span class="o">=</span> <span class="p">[(</span><span class="n">plus</span><span class="p">,</span> <span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="mi">8</span><span class="p">))</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)]</span>
|
||
|
||
<span class="c1"># Create queues</span>
|
||
<span class="n">task_queue</span> <span class="o">=</span> <span class="n">Queue</span><span class="p">()</span>
|
||
<span class="n">done_queue</span> <span class="o">=</span> <span class="n">Queue</span><span class="p">()</span>
|
||
|
||
<span class="c1"># Submit tasks</span>
|
||
<span class="k">for</span> <span class="n">task</span> <span class="ow">in</span> <span class="n">TASKS1</span><span class="p">:</span>
|
||
<span class="n">task_queue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">task</span><span class="p">)</span>
|
||
|
||
<span class="c1"># Start worker processes</span>
|
||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">NUMBER_OF_PROCESSES</span><span class="p">):</span>
|
||
<span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">worker</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">task_queue</span><span class="p">,</span> <span class="n">done_queue</span><span class="p">))</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
|
||
|
||
<span class="c1"># Get and print results</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'Unordered results:'</span><span class="p">)</span>
|
||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">TASKS1</span><span class="p">)):</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="se">\t</span><span class="s1">'</span><span class="p">,</span> <span class="n">done_queue</span><span class="o">.</span><span class="n">get</span><span class="p">())</span>
|
||
|
||
<span class="c1"># Add more tasks using `put()`</span>
|
||
<span class="k">for</span> <span class="n">task</span> <span class="ow">in</span> <span class="n">TASKS2</span><span class="p">:</span>
|
||
<span class="n">task_queue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">task</span><span class="p">)</span>
|
||
|
||
<span class="c1"># Get and print some more results</span>
|
||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">TASKS2</span><span class="p">)):</span>
|
||
<span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="se">\t</span><span class="s1">'</span><span class="p">,</span> <span class="n">done_queue</span><span class="o">.</span><span class="n">get</span><span class="p">())</span>
|
||
|
||
<span class="c1"># Tell child processes to stop</span>
|
||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">NUMBER_OF_PROCESSES</span><span class="p">):</span>
|
||
<span class="n">task_queue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'STOP'</span><span class="p">)</span>
|
||
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="n">freeze_support</span><span class="p">()</span>
|
||
<span class="n">test</span><span class="p">()</span>
|
||
</pre></div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
|
||
<div class="sphinxsidebarwrapper">
|
||
<h3><a href="../contents.html">Table of Contents</a></h3>
|
||
<ul>
|
||
<li><a class="reference internal" href="#"><code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing</span></code> — Process-based parallelism</a><ul>
|
||
<li><a class="reference internal" href="#introduction">Introduction</a><ul>
|
||
<li><a class="reference internal" href="#the-process-class">The <code class="xref py py-class docutils literal notranslate"><span class="pre">Process</span></code> class</a></li>
|
||
<li><a class="reference internal" href="#contexts-and-start-methods">Contexts and start methods</a></li>
|
||
<li><a class="reference internal" href="#exchanging-objects-between-processes">Exchanging objects between processes</a></li>
|
||
<li><a class="reference internal" href="#synchronization-between-processes">Synchronization between processes</a></li>
|
||
<li><a class="reference internal" href="#sharing-state-between-processes">Sharing state between processes</a></li>
|
||
<li><a class="reference internal" href="#using-a-pool-of-workers">Using a pool of workers</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a class="reference internal" href="#reference">Reference</a><ul>
|
||
<li><a class="reference internal" href="#process-and-exceptions"><code class="xref py py-class docutils literal notranslate"><span class="pre">Process</span></code> and exceptions</a></li>
|
||
<li><a class="reference internal" href="#pipes-and-queues">Pipes and Queues</a></li>
|
||
<li><a class="reference internal" href="#miscellaneous">Miscellaneous</a></li>
|
||
<li><a class="reference internal" href="#connection-objects">Connection Objects</a></li>
|
||
<li><a class="reference internal" href="#synchronization-primitives">Synchronization primitives</a></li>
|
||
<li><a class="reference internal" href="#shared-ctypes-objects">Shared <code class="xref py py-mod docutils literal notranslate"><span class="pre">ctypes</span></code> Objects</a><ul>
|
||
<li><a class="reference internal" href="#module-multiprocessing.sharedctypes">The <code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing.sharedctypes</span></code> module</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a class="reference internal" href="#managers">Managers</a><ul>
|
||
<li><a class="reference internal" href="#customized-managers">Customized managers</a></li>
|
||
<li><a class="reference internal" href="#using-a-remote-manager">Using a remote manager</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a class="reference internal" href="#proxy-objects">Proxy Objects</a><ul>
|
||
<li><a class="reference internal" href="#cleanup">Cleanup</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a class="reference internal" href="#module-multiprocessing.pool">Process Pools</a></li>
|
||
<li><a class="reference internal" href="#module-multiprocessing.connection">Listeners and Clients</a><ul>
|
||
<li><a class="reference internal" href="#address-formats">Address Formats</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a class="reference internal" href="#authentication-keys">Authentication keys</a></li>
|
||
<li><a class="reference internal" href="#logging">Logging</a></li>
|
||
<li><a class="reference internal" href="#module-multiprocessing.dummy">The <code class="xref py py-mod docutils literal notranslate"><span class="pre">multiprocessing.dummy</span></code> module</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a class="reference internal" href="#programming-guidelines">Programming guidelines</a><ul>
|
||
<li><a class="reference internal" href="#all-start-methods">All start methods</a></li>
|
||
<li><a class="reference internal" href="#the-spawn-and-forkserver-start-methods">The <em>spawn</em> and <em>forkserver</em> start methods</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a class="reference internal" href="#examples">Examples</a></li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
|
||
<h4>Previous topic</h4>
|
||
<p class="topless"><a href="threading.html"
|
||
title="previous chapter"><code class="xref py py-mod docutils literal notranslate"><span class="pre">threading</span></code> — Thread-based parallelism</a></p>
|
||
<h4>Next topic</h4>
|
||
<p class="topless"><a href="concurrent.html"
|
||
title="next chapter">The <code class="xref py py-mod docutils literal notranslate"><span class="pre">concurrent</span></code> package</a></p>
|
||
<div role="note" aria-label="source link">
|
||
<h3>This Page</h3>
|
||
<ul class="this-page-menu">
|
||
<li><a href="../bugs.html">Report a Bug</a></li>
|
||
<li>
|
||
<a href="https://github.com/python/cpython/blob/3.7/Doc/library/multiprocessing.rst"
|
||
rel="nofollow">Show Source
|
||
</a>
|
||
</li>
|
||
</ul>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="clearer"></div>
|
||
</div>
|
||
<div class="related" role="navigation" aria-label="related navigation">
|
||
<h3>Navigation</h3>
|
||
<ul>
|
||
<li class="right" style="margin-right: 10px">
|
||
<a href="../genindex.html" title="General Index"
|
||
>index</a></li>
|
||
<li class="right" >
|
||
<a href="../py-modindex.html" title="Python Module Index"
|
||
>modules</a> |</li>
|
||
<li class="right" >
|
||
<a href="concurrent.html" title="The concurrent package"
|
||
>next</a> |</li>
|
||
<li class="right" >
|
||
<a href="threading.html" title="threading — Thread-based parallelism"
|
||
>previous</a> |</li>
|
||
<li><img src="../_static/py.png" alt=""
|
||
style="vertical-align: middle; margin-top: -1px"/></li>
|
||
<li><a href="https://www.python.org/">Python</a> »</li>
|
||
<li>
|
||
<span class="language_switcher_placeholder">en</span>
|
||
<span class="version_switcher_placeholder">3.7.4</span>
|
||
<a href="../index.html">Documentation </a> »
|
||
</li>
|
||
|
||
<li class="nav-item nav-item-1"><a href="index.html" >The Python Standard Library</a> »</li>
|
||
<li class="nav-item nav-item-2"><a href="concurrency.html" >Concurrent Execution</a> »</li>
|
||
<li class="right">
|
||
|
||
|
||
<div class="inline-search" style="display: none" role="search">
|
||
<form class="inline-search" action="../search.html" method="get">
|
||
<input placeholder="Quick search" type="text" name="q" />
|
||
<input type="submit" value="Go" />
|
||
<input type="hidden" name="check_keywords" value="yes" />
|
||
<input type="hidden" name="area" value="default" />
|
||
</form>
|
||
</div>
|
||
<script type="text/javascript">$('.inline-search').show(0);</script>
|
||
|
|
||
</li>
|
||
|
||
</ul>
|
||
</div>
|
||
<div class="footer">
|
||
© <a href="../copyright.html">Copyright</a> 2001-2019, Python Software Foundation.
|
||
<br />
|
||
The Python Software Foundation is a non-profit corporation.
|
||
<a href="https://www.python.org/psf/donations/">Please donate.</a>
|
||
<br />
|
||
Last updated on Jul 13, 2019.
|
||
<a href="../bugs.html">Found a bug</a>?
|
||
<br />
|
||
Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 2.0.1.
|
||
</div>
|
||
|
||
</body>
|
||
</html> |