770 lines
70 KiB
HTML
770 lines
70 KiB
HTML
|
|
|||
|
|
<!DOCTYPE html>
|
|||
|
|
|
|||
|
|
<html xmlns="http://www.w3.org/1999/xhtml">
|
|||
|
|
<head>
|
|||
|
|
<meta charset="utf-8" />
|
|||
|
|
<title>HOWTO Fetch Internet Resources Using The urllib Package — Python 3.7.4 documentation</title>
|
|||
|
|
<link rel="stylesheet" href="../_static/pydoctheme.css" type="text/css" />
|
|||
|
|
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
|
|||
|
|
|
|||
|
|
<script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
|
|||
|
|
<script type="text/javascript" src="../_static/jquery.js"></script>
|
|||
|
|
<script type="text/javascript" src="../_static/underscore.js"></script>
|
|||
|
|
<script type="text/javascript" src="../_static/doctools.js"></script>
|
|||
|
|
<script type="text/javascript" src="../_static/language_data.js"></script>
|
|||
|
|
|
|||
|
|
<script type="text/javascript" src="../_static/sidebar.js"></script>
|
|||
|
|
|
|||
|
|
<link rel="search" type="application/opensearchdescription+xml"
|
|||
|
|
title="Search within Python 3.7.4 documentation"
|
|||
|
|
href="../_static/opensearch.xml"/>
|
|||
|
|
<link rel="author" title="About these documents" href="../about.html" />
|
|||
|
|
<link rel="index" title="Index" href="../genindex.html" />
|
|||
|
|
<link rel="search" title="Search" href="../search.html" />
|
|||
|
|
<link rel="copyright" title="Copyright" href="../copyright.html" />
|
|||
|
|
<link rel="next" title="Argparse Tutorial" href="argparse.html" />
|
|||
|
|
<link rel="prev" title="Unicode HOWTO" href="unicode.html" />
|
|||
|
|
<link rel="shortcut icon" type="image/png" href="../_static/py.png" />
|
|||
|
|
<link rel="canonical" href="https://docs.python.org/3/howto/urllib2.html" />
|
|||
|
|
|
|||
|
|
<script type="text/javascript" src="../_static/copybutton.js"></script>
|
|||
|
|
<script type="text/javascript" src="../_static/switchers.js"></script>
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
<style>
|
|||
|
|
@media only screen {
|
|||
|
|
table.full-width-table {
|
|||
|
|
width: 100%;
|
|||
|
|
}
|
|||
|
|
}
|
|||
|
|
</style>
|
|||
|
|
|
|||
|
|
|
|||
|
|
</head><body>
|
|||
|
|
|
|||
|
|
<div class="related" role="navigation" aria-label="related navigation">
|
|||
|
|
<h3>Navigation</h3>
|
|||
|
|
<ul>
|
|||
|
|
<li class="right" style="margin-right: 10px">
|
|||
|
|
<a href="../genindex.html" title="General Index"
|
|||
|
|
accesskey="I">index</a></li>
|
|||
|
|
<li class="right" >
|
|||
|
|
<a href="../py-modindex.html" title="Python Module Index"
|
|||
|
|
>modules</a> |</li>
|
|||
|
|
<li class="right" >
|
|||
|
|
<a href="argparse.html" title="Argparse Tutorial"
|
|||
|
|
accesskey="N">next</a> |</li>
|
|||
|
|
<li class="right" >
|
|||
|
|
<a href="unicode.html" title="Unicode HOWTO"
|
|||
|
|
accesskey="P">previous</a> |</li>
|
|||
|
|
<li><img src="../_static/py.png" alt=""
|
|||
|
|
style="vertical-align: middle; margin-top: -1px"/></li>
|
|||
|
|
<li><a href="https://www.python.org/">Python</a> »</li>
|
|||
|
|
<li>
|
|||
|
|
<span class="language_switcher_placeholder">en</span>
|
|||
|
|
<span class="version_switcher_placeholder">3.7.4</span>
|
|||
|
|
<a href="../index.html">Documentation </a> »
|
|||
|
|
</li>
|
|||
|
|
|
|||
|
|
<li class="nav-item nav-item-1"><a href="index.html" accesskey="U">Python HOWTOs</a> »</li>
|
|||
|
|
<li class="right">
|
|||
|
|
|
|||
|
|
|
|||
|
|
<div class="inline-search" style="display: none" role="search">
|
|||
|
|
<form class="inline-search" action="../search.html" method="get">
|
|||
|
|
<input placeholder="Quick search" type="text" name="q" />
|
|||
|
|
<input type="submit" value="Go" />
|
|||
|
|
<input type="hidden" name="check_keywords" value="yes" />
|
|||
|
|
<input type="hidden" name="area" value="default" />
|
|||
|
|
</form>
|
|||
|
|
</div>
|
|||
|
|
<script type="text/javascript">$('.inline-search').show(0);</script>
|
|||
|
|
|
|
|||
|
|
</li>
|
|||
|
|
|
|||
|
|
</ul>
|
|||
|
|
</div>
|
|||
|
|
|
|||
|
|
<div class="document">
|
|||
|
|
<div class="documentwrapper">
|
|||
|
|
<div class="bodywrapper">
|
|||
|
|
<div class="body" role="main">
|
|||
|
|
|
|||
|
|
<div class="section" id="howto-fetch-internet-resources-using-the-urllib-package">
|
|||
|
|
<span id="urllib-howto"></span><h1>HOWTO Fetch Internet Resources Using The urllib Package<a class="headerlink" href="#howto-fetch-internet-resources-using-the-urllib-package" title="Permalink to this headline">¶</a></h1>
|
|||
|
|
<dl class="field-list simple">
|
|||
|
|
<dt class="field-odd">Author</dt>
|
|||
|
|
<dd class="field-odd"><p><a class="reference external" href="http://www.voidspace.org.uk/python/index.shtml">Michael Foord</a></p>
|
|||
|
|
</dd>
|
|||
|
|
</dl>
|
|||
|
|
<div class="admonition note">
|
|||
|
|
<p class="admonition-title">Note</p>
|
|||
|
|
<p>There is a French translation of an earlier revision of this
|
|||
|
|
HOWTO, available at <a class="reference external" href="http://www.voidspace.org.uk/python/articles/urllib2_francais.shtml">urllib2 - Le Manuel manquant</a>.</p>
|
|||
|
|
</div>
|
|||
|
|
<div class="section" id="introduction">
|
|||
|
|
<h2>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h2>
|
|||
|
|
<div class="sidebar">
|
|||
|
|
<p class="sidebar-title">Related Articles</p>
|
|||
|
|
<p>You may also find useful the following article on fetching web resources
|
|||
|
|
with Python:</p>
|
|||
|
|
<ul>
|
|||
|
|
<li><p><a class="reference external" href="http://www.voidspace.org.uk/python/articles/authentication.shtml">Basic Authentication</a></p>
|
|||
|
|
<blockquote>
|
|||
|
|
<div><p>A tutorial on <em>Basic Authentication</em>, with examples in Python.</p>
|
|||
|
|
</div></blockquote>
|
|||
|
|
</li>
|
|||
|
|
</ul>
|
|||
|
|
</div>
|
|||
|
|
<p><strong>urllib.request</strong> is a Python module for fetching URLs
|
|||
|
|
(Uniform Resource Locators). It offers a very simple interface, in the form of
|
|||
|
|
the <em>urlopen</em> function. This is capable of fetching URLs using a variety of
|
|||
|
|
different protocols. It also offers a slightly more complex interface for
|
|||
|
|
handling common situations - like basic authentication, cookies, proxies and so
|
|||
|
|
on. These are provided by objects called handlers and openers.</p>
|
|||
|
|
<p>urllib.request supports fetching URLs for many “URL schemes” (identified by the string
|
|||
|
|
before the <code class="docutils literal notranslate"><span class="pre">":"</span></code> in URL - for example <code class="docutils literal notranslate"><span class="pre">"ftp"</span></code> is the URL scheme of
|
|||
|
|
<code class="docutils literal notranslate"><span class="pre">"ftp://python.org/"</span></code>) using their associated network protocols (e.g. FTP, HTTP).
|
|||
|
|
This tutorial focuses on the most common case, HTTP.</p>
|
|||
|
|
<p>For straightforward situations <em>urlopen</em> is very easy to use. But as soon as you
|
|||
|
|
encounter errors or non-trivial cases when opening HTTP URLs, you will need some
|
|||
|
|
understanding of the HyperText Transfer Protocol. The most comprehensive and
|
|||
|
|
authoritative reference to HTTP is <span class="target" id="index-0"></span><a class="rfc reference external" href="https://tools.ietf.org/html/rfc2616.html"><strong>RFC 2616</strong></a>. This is a technical document and
|
|||
|
|
not intended to be easy to read. This HOWTO aims to illustrate using <em>urllib</em>,
|
|||
|
|
with enough detail about HTTP to help you through. It is not intended to replace
|
|||
|
|
the <a class="reference internal" href="../library/urllib.request.html#module-urllib.request" title="urllib.request: Extensible library for opening URLs."><code class="xref py py-mod docutils literal notranslate"><span class="pre">urllib.request</span></code></a> docs, but is supplementary to them.</p>
|
|||
|
|
</div>
|
|||
|
|
<div class="section" id="fetching-urls">
|
|||
|
|
<h2>Fetching URLs<a class="headerlink" href="#fetching-urls" title="Permalink to this headline">¶</a></h2>
|
|||
|
|
<p>The simplest way to use urllib.request is as follows:</p>
|
|||
|
|
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">urllib.request</span>
|
|||
|
|
<span class="k">with</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="s1">'http://python.org/'</span><span class="p">)</span> <span class="k">as</span> <span class="n">response</span><span class="p">:</span>
|
|||
|
|
<span class="n">html</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
|
|||
|
|
</pre></div>
|
|||
|
|
</div>
|
|||
|
|
<p>If you wish to retrieve a resource via URL and store it in a temporary
|
|||
|
|
location, you can do so via the <a class="reference internal" href="../library/shutil.html#shutil.copyfileobj" title="shutil.copyfileobj"><code class="xref py py-func docutils literal notranslate"><span class="pre">shutil.copyfileobj()</span></code></a> and
|
|||
|
|
<a class="reference internal" href="../library/tempfile.html#tempfile.NamedTemporaryFile" title="tempfile.NamedTemporaryFile"><code class="xref py py-func docutils literal notranslate"><span class="pre">tempfile.NamedTemporaryFile()</span></code></a> functions:</p>
|
|||
|
|
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">shutil</span>
|
|||
|
|
<span class="kn">import</span> <span class="nn">tempfile</span>
|
|||
|
|
<span class="kn">import</span> <span class="nn">urllib.request</span>
|
|||
|
|
|
|||
|
|
<span class="k">with</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="s1">'http://python.org/'</span><span class="p">)</span> <span class="k">as</span> <span class="n">response</span><span class="p">:</span>
|
|||
|
|
<span class="k">with</span> <span class="n">tempfile</span><span class="o">.</span><span class="n">NamedTemporaryFile</span><span class="p">(</span><span class="n">delete</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span> <span class="k">as</span> <span class="n">tmp_file</span><span class="p">:</span>
|
|||
|
|
<span class="n">shutil</span><span class="o">.</span><span class="n">copyfileobj</span><span class="p">(</span><span class="n">response</span><span class="p">,</span> <span class="n">tmp_file</span><span class="p">)</span>
|
|||
|
|
|
|||
|
|
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">tmp_file</span><span class="o">.</span><span class="n">name</span><span class="p">)</span> <span class="k">as</span> <span class="n">html</span><span class="p">:</span>
|
|||
|
|
<span class="k">pass</span>
|
|||
|
|
</pre></div>
|
|||
|
|
</div>
|
|||
|
|
<p>Many uses of urllib will be that simple (note that instead of an ‘http:’ URL we
|
|||
|
|
could have used a URL starting with ‘ftp:’, ‘file:’, etc.). However, it’s the
|
|||
|
|
purpose of this tutorial to explain the more complicated cases, concentrating on
|
|||
|
|
HTTP.</p>
|
|||
|
|
<p>HTTP is based on requests and responses - the client makes requests and servers
|
|||
|
|
send responses. urllib.request mirrors this with a <code class="docutils literal notranslate"><span class="pre">Request</span></code> object which represents
|
|||
|
|
the HTTP request you are making. In its simplest form you create a Request
|
|||
|
|
object that specifies the URL you want to fetch. Calling <code class="docutils literal notranslate"><span class="pre">urlopen</span></code> with this
|
|||
|
|
Request object returns a response object for the URL requested. This response is
|
|||
|
|
a file-like object, which means you can for example call <code class="docutils literal notranslate"><span class="pre">.read()</span></code> on the
|
|||
|
|
response:</p>
|
|||
|
|
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">urllib.request</span>
|
|||
|
|
|
|||
|
|
<span class="n">req</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="s1">'http://www.voidspace.org.uk'</span><span class="p">)</span>
|
|||
|
|
<span class="k">with</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">req</span><span class="p">)</span> <span class="k">as</span> <span class="n">response</span><span class="p">:</span>
|
|||
|
|
<span class="n">the_page</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
|
|||
|
|
</pre></div>
|
|||
|
|
</div>
|
|||
|
|
<p>Note that urllib.request makes use of the same Request interface to handle all URL
|
|||
|
|
schemes. For example, you can make an FTP request like so:</p>
|
|||
|
|
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="n">req</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="s1">'ftp://example.com/'</span><span class="p">)</span>
|
|||
|
|
</pre></div>
|
|||
|
|
</div>
|
|||
|
|
<p>In the case of HTTP, there are two extra things that Request objects allow you
|
|||
|
|
to do: First, you can pass data to be sent to the server. Second, you can pass
|
|||
|
|
extra information (“metadata”) <em>about</em> the data or the about request itself, to
|
|||
|
|
the server - this information is sent as HTTP “headers”. Let’s look at each of
|
|||
|
|
these in turn.</p>
|
|||
|
|
<div class="section" id="data">
|
|||
|
|
<h3>Data<a class="headerlink" href="#data" title="Permalink to this headline">¶</a></h3>
|
|||
|
|
<p>Sometimes you want to send data to a URL (often the URL will refer to a CGI
|
|||
|
|
(Common Gateway Interface) script or other web application). With HTTP,
|
|||
|
|
this is often done using what’s known as a <strong>POST</strong> request. This is often what
|
|||
|
|
your browser does when you submit a HTML form that you filled in on the web. Not
|
|||
|
|
all POSTs have to come from forms: you can use a POST to transmit arbitrary data
|
|||
|
|
to your own application. In the common case of HTML forms, the data needs to be
|
|||
|
|
encoded in a standard way, and then passed to the Request object as the <code class="docutils literal notranslate"><span class="pre">data</span></code>
|
|||
|
|
argument. The encoding is done using a function from the <a class="reference internal" href="../library/urllib.parse.html#module-urllib.parse" title="urllib.parse: Parse URLs into or assemble them from components."><code class="xref py py-mod docutils literal notranslate"><span class="pre">urllib.parse</span></code></a>
|
|||
|
|
library.</p>
|
|||
|
|
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">urllib.parse</span>
|
|||
|
|
<span class="kn">import</span> <span class="nn">urllib.request</span>
|
|||
|
|
|
|||
|
|
<span class="n">url</span> <span class="o">=</span> <span class="s1">'http://www.someserver.com/cgi-bin/register.cgi'</span>
|
|||
|
|
<span class="n">values</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'name'</span> <span class="p">:</span> <span class="s1">'Michael Foord'</span><span class="p">,</span>
|
|||
|
|
<span class="s1">'location'</span> <span class="p">:</span> <span class="s1">'Northampton'</span><span class="p">,</span>
|
|||
|
|
<span class="s1">'language'</span> <span class="p">:</span> <span class="s1">'Python'</span> <span class="p">}</span>
|
|||
|
|
|
|||
|
|
<span class="n">data</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">parse</span><span class="o">.</span><span class="n">urlencode</span><span class="p">(</span><span class="n">values</span><span class="p">)</span>
|
|||
|
|
<span class="n">data</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">)</span> <span class="c1"># data should be bytes</span>
|
|||
|
|
<span class="n">req</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">data</span><span class="p">)</span>
|
|||
|
|
<span class="k">with</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">req</span><span class="p">)</span> <span class="k">as</span> <span class="n">response</span><span class="p">:</span>
|
|||
|
|
<span class="n">the_page</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
|
|||
|
|
</pre></div>
|
|||
|
|
</div>
|
|||
|
|
<p>Note that other encodings are sometimes required (e.g. for file upload from HTML
|
|||
|
|
forms - see <a class="reference external" href="https://www.w3.org/TR/REC-html40/interact/forms.html#h-17.13">HTML Specification, Form Submission</a> for more
|
|||
|
|
details).</p>
|
|||
|
|
<p>If you do not pass the <code class="docutils literal notranslate"><span class="pre">data</span></code> argument, urllib uses a <strong>GET</strong> request. One
|
|||
|
|
way in which GET and POST requests differ is that POST requests often have
|
|||
|
|
“side-effects”: they change the state of the system in some way (for example by
|
|||
|
|
placing an order with the website for a hundredweight of tinned spam to be
|
|||
|
|
delivered to your door). Though the HTTP standard makes it clear that POSTs are
|
|||
|
|
intended to <em>always</em> cause side-effects, and GET requests <em>never</em> to cause
|
|||
|
|
side-effects, nothing prevents a GET request from having side-effects, nor a
|
|||
|
|
POST requests from having no side-effects. Data can also be passed in an HTTP
|
|||
|
|
GET request by encoding it in the URL itself.</p>
|
|||
|
|
<p>This is done as follows:</p>
|
|||
|
|
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">urllib.request</span>
|
|||
|
|
<span class="gp">>>> </span><span class="kn">import</span> <span class="nn">urllib.parse</span>
|
|||
|
|
<span class="gp">>>> </span><span class="n">data</span> <span class="o">=</span> <span class="p">{}</span>
|
|||
|
|
<span class="gp">>>> </span><span class="n">data</span><span class="p">[</span><span class="s1">'name'</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'Somebody Here'</span>
|
|||
|
|
<span class="gp">>>> </span><span class="n">data</span><span class="p">[</span><span class="s1">'location'</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'Northampton'</span>
|
|||
|
|
<span class="gp">>>> </span><span class="n">data</span><span class="p">[</span><span class="s1">'language'</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'Python'</span>
|
|||
|
|
<span class="gp">>>> </span><span class="n">url_values</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">parse</span><span class="o">.</span><span class="n">urlencode</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
|
|||
|
|
<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">url_values</span><span class="p">)</span> <span class="c1"># The order may differ from below. </span>
|
|||
|
|
<span class="go">name=Somebody+Here&language=Python&location=Northampton</span>
|
|||
|
|
<span class="gp">>>> </span><span class="n">url</span> <span class="o">=</span> <span class="s1">'http://www.example.com/example.cgi'</span>
|
|||
|
|
<span class="gp">>>> </span><span class="n">full_url</span> <span class="o">=</span> <span class="n">url</span> <span class="o">+</span> <span class="s1">'?'</span> <span class="o">+</span> <span class="n">url_values</span>
|
|||
|
|
<span class="gp">>>> </span><span class="n">data</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">full_url</span><span class="p">)</span>
|
|||
|
|
</pre></div>
|
|||
|
|
</div>
|
|||
|
|
<p>Notice that the full URL is created by adding a <code class="docutils literal notranslate"><span class="pre">?</span></code> to the URL, followed by
|
|||
|
|
the encoded values.</p>
|
|||
|
|
</div>
|
|||
|
|
<div class="section" id="headers">
|
|||
|
|
<h3>Headers<a class="headerlink" href="#headers" title="Permalink to this headline">¶</a></h3>
|
|||
|
|
<p>We’ll discuss here one particular HTTP header, to illustrate how to add headers
|
|||
|
|
to your HTTP request.</p>
|
|||
|
|
<p>Some websites <a class="footnote-reference brackets" href="#id8" id="id1">1</a> dislike being browsed by programs, or send different versions
|
|||
|
|
to different browsers <a class="footnote-reference brackets" href="#id9" id="id2">2</a>. By default urllib identifies itself as
|
|||
|
|
<code class="docutils literal notranslate"><span class="pre">Python-urllib/x.y</span></code> (where <code class="docutils literal notranslate"><span class="pre">x</span></code> and <code class="docutils literal notranslate"><span class="pre">y</span></code> are the major and minor version
|
|||
|
|
numbers of the Python release,
|
|||
|
|
e.g. <code class="docutils literal notranslate"><span class="pre">Python-urllib/2.5</span></code>), which may confuse the site, or just plain
|
|||
|
|
not work. The way a browser identifies itself is through the
|
|||
|
|
<code class="docutils literal notranslate"><span class="pre">User-Agent</span></code> header <a class="footnote-reference brackets" href="#id10" id="id3">3</a>. When you create a Request object you can
|
|||
|
|
pass a dictionary of headers in. The following example makes the same
|
|||
|
|
request as above, but identifies itself as a version of Internet
|
|||
|
|
Explorer <a class="footnote-reference brackets" href="#id11" id="id4">4</a>.</p>
|
|||
|
|
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">urllib.parse</span>
|
|||
|
|
<span class="kn">import</span> <span class="nn">urllib.request</span>
|
|||
|
|
|
|||
|
|
<span class="n">url</span> <span class="o">=</span> <span class="s1">'http://www.someserver.com/cgi-bin/register.cgi'</span>
|
|||
|
|
<span class="n">user_agent</span> <span class="o">=</span> <span class="s1">'Mozilla/5.0 (Windows NT 6.1; Win64; x64)'</span>
|
|||
|
|
<span class="n">values</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'name'</span><span class="p">:</span> <span class="s1">'Michael Foord'</span><span class="p">,</span>
|
|||
|
|
<span class="s1">'location'</span><span class="p">:</span> <span class="s1">'Northampton'</span><span class="p">,</span>
|
|||
|
|
<span class="s1">'language'</span><span class="p">:</span> <span class="s1">'Python'</span> <span class="p">}</span>
|
|||
|
|
<span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'User-Agent'</span><span class="p">:</span> <span class="n">user_agent</span><span class="p">}</span>
|
|||
|
|
|
|||
|
|
<span class="n">data</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">parse</span><span class="o">.</span><span class="n">urlencode</span><span class="p">(</span><span class="n">values</span><span class="p">)</span>
|
|||
|
|
<span class="n">data</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">)</span>
|
|||
|
|
<span class="n">req</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">data</span><span class="p">,</span> <span class="n">headers</span><span class="p">)</span>
|
|||
|
|
<span class="k">with</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">req</span><span class="p">)</span> <span class="k">as</span> <span class="n">response</span><span class="p">:</span>
|
|||
|
|
<span class="n">the_page</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
|
|||
|
|
</pre></div>
|
|||
|
|
</div>
|
|||
|
|
<p>The response also has two useful methods. See the section on <a class="reference internal" href="#info-and-geturl">info and geturl</a>
|
|||
|
|
which comes after we have a look at what happens when things go wrong.</p>
|
|||
|
|
</div>
|
|||
|
|
</div>
|
|||
|
|
<div class="section" id="handling-exceptions">
|
|||
|
|
<h2>Handling Exceptions<a class="headerlink" href="#handling-exceptions" title="Permalink to this headline">¶</a></h2>
|
|||
|
|
<p><em>urlopen</em> raises <code class="xref py py-exc docutils literal notranslate"><span class="pre">URLError</span></code> when it cannot handle a response (though as
|
|||
|
|
usual with Python APIs, built-in exceptions such as <a class="reference internal" href="../library/exceptions.html#ValueError" title="ValueError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">ValueError</span></code></a>,
|
|||
|
|
<a class="reference internal" href="../library/exceptions.html#TypeError" title="TypeError"><code class="xref py py-exc docutils literal notranslate"><span class="pre">TypeError</span></code></a> etc. may also be raised).</p>
|
|||
|
|
<p><code class="xref py py-exc docutils literal notranslate"><span class="pre">HTTPError</span></code> is the subclass of <code class="xref py py-exc docutils literal notranslate"><span class="pre">URLError</span></code> raised in the specific case of
|
|||
|
|
HTTP URLs.</p>
|
|||
|
|
<p>The exception classes are exported from the <a class="reference internal" href="../library/urllib.error.html#module-urllib.error" title="urllib.error: Exception classes raised by urllib.request."><code class="xref py py-mod docutils literal notranslate"><span class="pre">urllib.error</span></code></a> module.</p>
|
|||
|
|
<div class="section" id="urlerror">
|
|||
|
|
<h3>URLError<a class="headerlink" href="#urlerror" title="Permalink to this headline">¶</a></h3>
|
|||
|
|
<p>Often, URLError is raised because there is no network connection (no route to
|
|||
|
|
the specified server), or the specified server doesn’t exist. In this case, the
|
|||
|
|
exception raised will have a ‘reason’ attribute, which is a tuple containing an
|
|||
|
|
error code and a text error message.</p>
|
|||
|
|
<p>e.g.</p>
|
|||
|
|
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">req</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="s1">'http://www.pretend_server.org'</span><span class="p">)</span>
|
|||
|
|
<span class="gp">>>> </span><span class="k">try</span><span class="p">:</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">req</span><span class="p">)</span>
|
|||
|
|
<span class="gp">... </span><span class="k">except</span> <span class="n">urllib</span><span class="o">.</span><span class="n">error</span><span class="o">.</span><span class="n">URLError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
|
|||
|
|
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="n">reason</span><span class="p">)</span>
|
|||
|
|
<span class="gp">...</span>
|
|||
|
|
<span class="go">(4, 'getaddrinfo failed')</span>
|
|||
|
|
</pre></div>
|
|||
|
|
</div>
|
|||
|
|
</div>
|
|||
|
|
<div class="section" id="httperror">
|
|||
|
|
<h3>HTTPError<a class="headerlink" href="#httperror" title="Permalink to this headline">¶</a></h3>
|
|||
|
|
<p>Every HTTP response from the server contains a numeric “status code”. Sometimes
|
|||
|
|
the status code indicates that the server is unable to fulfil the request. The
|
|||
|
|
default handlers will handle some of these responses for you (for example, if
|
|||
|
|
the response is a “redirection” that requests the client fetch the document from
|
|||
|
|
a different URL, urllib will handle that for you). For those it can’t handle,
|
|||
|
|
urlopen will raise an <code class="xref py py-exc docutils literal notranslate"><span class="pre">HTTPError</span></code>. Typical errors include ‘404’ (page not
|
|||
|
|
found), ‘403’ (request forbidden), and ‘401’ (authentication required).</p>
|
|||
|
|
<p>See section 10 of <span class="target" id="index-1"></span><a class="rfc reference external" href="https://tools.ietf.org/html/rfc2616.html"><strong>RFC 2616</strong></a> for a reference on all the HTTP error codes.</p>
|
|||
|
|
<p>The <code class="xref py py-exc docutils literal notranslate"><span class="pre">HTTPError</span></code> instance raised will have an integer ‘code’ attribute, which
|
|||
|
|
corresponds to the error sent by the server.</p>
|
|||
|
|
<div class="section" id="error-codes">
|
|||
|
|
<h4>Error Codes<a class="headerlink" href="#error-codes" title="Permalink to this headline">¶</a></h4>
|
|||
|
|
<p>Because the default handlers handle redirects (codes in the 300 range), and
|
|||
|
|
codes in the 100–299 range indicate success, you will usually only see error
|
|||
|
|
codes in the 400–599 range.</p>
|
|||
|
|
<p><a class="reference internal" href="../library/http.server.html#http.server.BaseHTTPRequestHandler.responses" title="http.server.BaseHTTPRequestHandler.responses"><code class="xref py py-attr docutils literal notranslate"><span class="pre">http.server.BaseHTTPRequestHandler.responses</span></code></a> is a useful dictionary of
|
|||
|
|
response codes in that shows all the response codes used by <span class="target" id="index-2"></span><a class="rfc reference external" href="https://tools.ietf.org/html/rfc2616.html"><strong>RFC 2616</strong></a>. The
|
|||
|
|
dictionary is reproduced here for convenience</p>
|
|||
|
|
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Table mapping response codes to messages; entries have the</span>
|
|||
|
|
<span class="c1"># form {code: (shortmessage, longmessage)}.</span>
|
|||
|
|
<span class="n">responses</span> <span class="o">=</span> <span class="p">{</span>
|
|||
|
|
<span class="mi">100</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Continue'</span><span class="p">,</span> <span class="s1">'Request received, please continue'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">101</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Switching Protocols'</span><span class="p">,</span>
|
|||
|
|
<span class="s1">'Switching to new protocol; obey Upgrade header'</span><span class="p">),</span>
|
|||
|
|
|
|||
|
|
<span class="mi">200</span><span class="p">:</span> <span class="p">(</span><span class="s1">'OK'</span><span class="p">,</span> <span class="s1">'Request fulfilled, document follows'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">201</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Created'</span><span class="p">,</span> <span class="s1">'Document created, URL follows'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">202</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Accepted'</span><span class="p">,</span>
|
|||
|
|
<span class="s1">'Request accepted, processing continues off-line'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">203</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Non-Authoritative Information'</span><span class="p">,</span> <span class="s1">'Request fulfilled from cache'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">204</span><span class="p">:</span> <span class="p">(</span><span class="s1">'No Content'</span><span class="p">,</span> <span class="s1">'Request fulfilled, nothing follows'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">205</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Reset Content'</span><span class="p">,</span> <span class="s1">'Clear input form for further input.'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">206</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Partial Content'</span><span class="p">,</span> <span class="s1">'Partial content follows.'</span><span class="p">),</span>
|
|||
|
|
|
|||
|
|
<span class="mi">300</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Multiple Choices'</span><span class="p">,</span>
|
|||
|
|
<span class="s1">'Object has several resources -- see URI list'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">301</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Moved Permanently'</span><span class="p">,</span> <span class="s1">'Object moved permanently -- see URI list'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">302</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Found'</span><span class="p">,</span> <span class="s1">'Object moved temporarily -- see URI list'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">303</span><span class="p">:</span> <span class="p">(</span><span class="s1">'See Other'</span><span class="p">,</span> <span class="s1">'Object moved -- see Method and URL list'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">304</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Not Modified'</span><span class="p">,</span>
|
|||
|
|
<span class="s1">'Document has not changed since given time'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">305</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Use Proxy'</span><span class="p">,</span>
|
|||
|
|
<span class="s1">'You must use proxy specified in Location to access this '</span>
|
|||
|
|
<span class="s1">'resource.'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">307</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Temporary Redirect'</span><span class="p">,</span>
|
|||
|
|
<span class="s1">'Object moved temporarily -- see URI list'</span><span class="p">),</span>
|
|||
|
|
|
|||
|
|
<span class="mi">400</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Bad Request'</span><span class="p">,</span>
|
|||
|
|
<span class="s1">'Bad request syntax or unsupported method'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">401</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Unauthorized'</span><span class="p">,</span>
|
|||
|
|
<span class="s1">'No permission -- see authorization schemes'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">402</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Payment Required'</span><span class="p">,</span>
|
|||
|
|
<span class="s1">'No payment -- see charging schemes'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">403</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Forbidden'</span><span class="p">,</span>
|
|||
|
|
<span class="s1">'Request forbidden -- authorization will not help'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">404</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Not Found'</span><span class="p">,</span> <span class="s1">'Nothing matches the given URI'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">405</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Method Not Allowed'</span><span class="p">,</span>
|
|||
|
|
<span class="s1">'Specified method is invalid for this server.'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">406</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Not Acceptable'</span><span class="p">,</span> <span class="s1">'URI not available in preferred format.'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">407</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Proxy Authentication Required'</span><span class="p">,</span> <span class="s1">'You must authenticate with '</span>
|
|||
|
|
<span class="s1">'this proxy before proceeding.'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">408</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Request Timeout'</span><span class="p">,</span> <span class="s1">'Request timed out; try again later.'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">409</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Conflict'</span><span class="p">,</span> <span class="s1">'Request conflict.'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">410</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Gone'</span><span class="p">,</span>
|
|||
|
|
<span class="s1">'URI no longer exists and has been permanently removed.'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">411</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Length Required'</span><span class="p">,</span> <span class="s1">'Client must specify Content-Length.'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">412</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Precondition Failed'</span><span class="p">,</span> <span class="s1">'Precondition in headers is false.'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">413</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Request Entity Too Large'</span><span class="p">,</span> <span class="s1">'Entity is too large.'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">414</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Request-URI Too Long'</span><span class="p">,</span> <span class="s1">'URI is too long.'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">415</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Unsupported Media Type'</span><span class="p">,</span> <span class="s1">'Entity body in unsupported format.'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">416</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Requested Range Not Satisfiable'</span><span class="p">,</span>
|
|||
|
|
<span class="s1">'Cannot satisfy request range.'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">417</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Expectation Failed'</span><span class="p">,</span>
|
|||
|
|
<span class="s1">'Expect condition could not be satisfied.'</span><span class="p">),</span>
|
|||
|
|
|
|||
|
|
<span class="mi">500</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Internal Server Error'</span><span class="p">,</span> <span class="s1">'Server got itself in trouble'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">501</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Not Implemented'</span><span class="p">,</span>
|
|||
|
|
<span class="s1">'Server does not support this operation'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">502</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Bad Gateway'</span><span class="p">,</span> <span class="s1">'Invalid responses from another server/proxy.'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">503</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Service Unavailable'</span><span class="p">,</span>
|
|||
|
|
<span class="s1">'The server cannot process the request due to a high load'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">504</span><span class="p">:</span> <span class="p">(</span><span class="s1">'Gateway Timeout'</span><span class="p">,</span>
|
|||
|
|
<span class="s1">'The gateway server did not receive a timely response'</span><span class="p">),</span>
|
|||
|
|
<span class="mi">505</span><span class="p">:</span> <span class="p">(</span><span class="s1">'HTTP Version Not Supported'</span><span class="p">,</span> <span class="s1">'Cannot fulfill request.'</span><span class="p">),</span>
|
|||
|
|
<span class="p">}</span>
|
|||
|
|
</pre></div>
|
|||
|
|
</div>
|
|||
|
|
<p>When an error is raised the server responds by returning an HTTP error code
|
|||
|
|
<em>and</em> an error page. You can use the <code class="xref py py-exc docutils literal notranslate"><span class="pre">HTTPError</span></code> instance as a response on the
|
|||
|
|
page returned. This means that as well as the code attribute, it also has read,
|
|||
|
|
geturl, and info, methods as returned by the <code class="docutils literal notranslate"><span class="pre">urllib.response</span></code> module:</p>
|
|||
|
|
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">req</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="s1">'http://www.python.org/fish.html'</span><span class="p">)</span>
|
|||
|
|
<span class="gp">>>> </span><span class="k">try</span><span class="p">:</span>
|
|||
|
|
<span class="gp">... </span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">req</span><span class="p">)</span>
|
|||
|
|
<span class="gp">... </span><span class="k">except</span> <span class="n">urllib</span><span class="o">.</span><span class="n">error</span><span class="o">.</span><span class="n">HTTPError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
|
|||
|
|
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="n">code</span><span class="p">)</span>
|
|||
|
|
<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="n">read</span><span class="p">())</span>
|
|||
|
|
<span class="gp">...</span>
|
|||
|
|
<span class="go">404</span>
|
|||
|
|
<span class="go">b'<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"</span>
|
|||
|
|
<span class="go"> "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\n\n\n<html</span>
|
|||
|
|
<span class="go"> ...</span>
|
|||
|
|
<span class="go"> <title>Page Not Found</title>\n</span>
|
|||
|
|
<span class="go"> ...</span>
|
|||
|
|
</pre></div>
|
|||
|
|
</div>
|
|||
|
|
</div>
|
|||
|
|
</div>
|
|||
|
|
<div class="section" id="wrapping-it-up">
|
|||
|
|
<h3>Wrapping it Up<a class="headerlink" href="#wrapping-it-up" title="Permalink to this headline">¶</a></h3>
|
|||
|
|
<p>So if you want to be prepared for <code class="xref py py-exc docutils literal notranslate"><span class="pre">HTTPError</span></code> <em>or</em> <code class="xref py py-exc docutils literal notranslate"><span class="pre">URLError</span></code> there are two
|
|||
|
|
basic approaches. I prefer the second approach.</p>
|
|||
|
|
<div class="section" id="number-1">
|
|||
|
|
<h4>Number 1<a class="headerlink" href="#number-1" title="Permalink to this headline">¶</a></h4>
|
|||
|
|
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">urllib.request</span> <span class="k">import</span> <span class="n">Request</span><span class="p">,</span> <span class="n">urlopen</span>
|
|||
|
|
<span class="kn">from</span> <span class="nn">urllib.error</span> <span class="k">import</span> <span class="n">URLError</span><span class="p">,</span> <span class="n">HTTPError</span>
|
|||
|
|
<span class="n">req</span> <span class="o">=</span> <span class="n">Request</span><span class="p">(</span><span class="n">someurl</span><span class="p">)</span>
|
|||
|
|
<span class="k">try</span><span class="p">:</span>
|
|||
|
|
<span class="n">response</span> <span class="o">=</span> <span class="n">urlopen</span><span class="p">(</span><span class="n">req</span><span class="p">)</span>
|
|||
|
|
<span class="k">except</span> <span class="n">HTTPError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
|
|||
|
|
<span class="nb">print</span><span class="p">(</span><span class="s1">'The server couldn</span><span class="se">\'</span><span class="s1">t fulfill the request.'</span><span class="p">)</span>
|
|||
|
|
<span class="nb">print</span><span class="p">(</span><span class="s1">'Error code: '</span><span class="p">,</span> <span class="n">e</span><span class="o">.</span><span class="n">code</span><span class="p">)</span>
|
|||
|
|
<span class="k">except</span> <span class="n">URLError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
|
|||
|
|
<span class="nb">print</span><span class="p">(</span><span class="s1">'We failed to reach a server.'</span><span class="p">)</span>
|
|||
|
|
<span class="nb">print</span><span class="p">(</span><span class="s1">'Reason: '</span><span class="p">,</span> <span class="n">e</span><span class="o">.</span><span class="n">reason</span><span class="p">)</span>
|
|||
|
|
<span class="k">else</span><span class="p">:</span>
|
|||
|
|
<span class="c1"># everything is fine</span>
|
|||
|
|
</pre></div>
|
|||
|
|
</div>
|
|||
|
|
<div class="admonition note">
|
|||
|
|
<p class="admonition-title">Note</p>
|
|||
|
|
<p>The <code class="docutils literal notranslate"><span class="pre">except</span> <span class="pre">HTTPError</span></code> <em>must</em> come first, otherwise <code class="docutils literal notranslate"><span class="pre">except</span> <span class="pre">URLError</span></code>
|
|||
|
|
will <em>also</em> catch an <code class="xref py py-exc docutils literal notranslate"><span class="pre">HTTPError</span></code>.</p>
|
|||
|
|
</div>
|
|||
|
|
</div>
|
|||
|
|
<div class="section" id="number-2">
|
|||
|
|
<h4>Number 2<a class="headerlink" href="#number-2" title="Permalink to this headline">¶</a></h4>
|
|||
|
|
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">urllib.request</span> <span class="k">import</span> <span class="n">Request</span><span class="p">,</span> <span class="n">urlopen</span>
|
|||
|
|
<span class="kn">from</span> <span class="nn">urllib.error</span> <span class="k">import</span> <span class="n">URLError</span>
|
|||
|
|
<span class="n">req</span> <span class="o">=</span> <span class="n">Request</span><span class="p">(</span><span class="n">someurl</span><span class="p">)</span>
|
|||
|
|
<span class="k">try</span><span class="p">:</span>
|
|||
|
|
<span class="n">response</span> <span class="o">=</span> <span class="n">urlopen</span><span class="p">(</span><span class="n">req</span><span class="p">)</span>
|
|||
|
|
<span class="k">except</span> <span class="n">URLError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
|
|||
|
|
<span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">e</span><span class="p">,</span> <span class="s1">'reason'</span><span class="p">):</span>
|
|||
|
|
<span class="nb">print</span><span class="p">(</span><span class="s1">'We failed to reach a server.'</span><span class="p">)</span>
|
|||
|
|
<span class="nb">print</span><span class="p">(</span><span class="s1">'Reason: '</span><span class="p">,</span> <span class="n">e</span><span class="o">.</span><span class="n">reason</span><span class="p">)</span>
|
|||
|
|
<span class="k">elif</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">e</span><span class="p">,</span> <span class="s1">'code'</span><span class="p">):</span>
|
|||
|
|
<span class="nb">print</span><span class="p">(</span><span class="s1">'The server couldn</span><span class="se">\'</span><span class="s1">t fulfill the request.'</span><span class="p">)</span>
|
|||
|
|
<span class="nb">print</span><span class="p">(</span><span class="s1">'Error code: '</span><span class="p">,</span> <span class="n">e</span><span class="o">.</span><span class="n">code</span><span class="p">)</span>
|
|||
|
|
<span class="k">else</span><span class="p">:</span>
|
|||
|
|
<span class="c1"># everything is fine</span>
|
|||
|
|
</pre></div>
|
|||
|
|
</div>
|
|||
|
|
</div>
|
|||
|
|
</div>
|
|||
|
|
</div>
|
|||
|
|
<div class="section" id="info-and-geturl">
|
|||
|
|
<h2>info and geturl<a class="headerlink" href="#info-and-geturl" title="Permalink to this headline">¶</a></h2>
|
|||
|
|
<p>The response returned by urlopen (or the <code class="xref py py-exc docutils literal notranslate"><span class="pre">HTTPError</span></code> instance) has two
|
|||
|
|
useful methods <code class="xref py py-meth docutils literal notranslate"><span class="pre">info()</span></code> and <code class="xref py py-meth docutils literal notranslate"><span class="pre">geturl()</span></code> and is defined in the module
|
|||
|
|
<a class="reference internal" href="../library/urllib.request.html#module-urllib.response" title="urllib.response: Response classes used by urllib."><code class="xref py py-mod docutils literal notranslate"><span class="pre">urllib.response</span></code></a>..</p>
|
|||
|
|
<p><strong>geturl</strong> - this returns the real URL of the page fetched. This is useful
|
|||
|
|
because <code class="docutils literal notranslate"><span class="pre">urlopen</span></code> (or the opener object used) may have followed a
|
|||
|
|
redirect. The URL of the page fetched may not be the same as the URL requested.</p>
|
|||
|
|
<p><strong>info</strong> - this returns a dictionary-like object that describes the page
|
|||
|
|
fetched, particularly the headers sent by the server. It is currently an
|
|||
|
|
<code class="xref py py-class docutils literal notranslate"><span class="pre">http.client.HTTPMessage</span></code> instance.</p>
|
|||
|
|
<p>Typical headers include ‘Content-length’, ‘Content-type’, and so on. See the
|
|||
|
|
<a class="reference external" href="http://jkorpela.fi/http.html">Quick Reference to HTTP Headers</a>
|
|||
|
|
for a useful listing of HTTP headers with brief explanations of their meaning
|
|||
|
|
and use.</p>
|
|||
|
|
</div>
|
|||
|
|
<div class="section" id="openers-and-handlers">
|
|||
|
|
<h2>Openers and Handlers<a class="headerlink" href="#openers-and-handlers" title="Permalink to this headline">¶</a></h2>
|
|||
|
|
<p>When you fetch a URL you use an opener (an instance of the perhaps
|
|||
|
|
confusingly-named <a class="reference internal" href="../library/urllib.request.html#urllib.request.OpenerDirector" title="urllib.request.OpenerDirector"><code class="xref py py-class docutils literal notranslate"><span class="pre">urllib.request.OpenerDirector</span></code></a>). Normally we have been using
|
|||
|
|
the default opener - via <code class="docutils literal notranslate"><span class="pre">urlopen</span></code> - but you can create custom
|
|||
|
|
openers. Openers use handlers. All the “heavy lifting” is done by the
|
|||
|
|
handlers. Each handler knows how to open URLs for a particular URL scheme (http,
|
|||
|
|
ftp, etc.), or how to handle an aspect of URL opening, for example HTTP
|
|||
|
|
redirections or HTTP cookies.</p>
|
|||
|
|
<p>You will want to create openers if you want to fetch URLs with specific handlers
|
|||
|
|
installed, for example to get an opener that handles cookies, or to get an
|
|||
|
|
opener that does not handle redirections.</p>
|
|||
|
|
<p>To create an opener, instantiate an <code class="docutils literal notranslate"><span class="pre">OpenerDirector</span></code>, and then call
|
|||
|
|
<code class="docutils literal notranslate"><span class="pre">.add_handler(some_handler_instance)</span></code> repeatedly.</p>
|
|||
|
|
<p>Alternatively, you can use <code class="docutils literal notranslate"><span class="pre">build_opener</span></code>, which is a convenience function for
|
|||
|
|
creating opener objects with a single function call. <code class="docutils literal notranslate"><span class="pre">build_opener</span></code> adds
|
|||
|
|
several handlers by default, but provides a quick way to add more and/or
|
|||
|
|
override the default handlers.</p>
|
|||
|
|
<p>Other sorts of handlers you might want to can handle proxies, authentication,
|
|||
|
|
and other common but slightly specialised situations.</p>
|
|||
|
|
<p><code class="docutils literal notranslate"><span class="pre">install_opener</span></code> can be used to make an <code class="docutils literal notranslate"><span class="pre">opener</span></code> object the (global) default
|
|||
|
|
opener. This means that calls to <code class="docutils literal notranslate"><span class="pre">urlopen</span></code> will use the opener you have
|
|||
|
|
installed.</p>
|
|||
|
|
<p>Opener objects have an <code class="docutils literal notranslate"><span class="pre">open</span></code> method, which can be called directly to fetch
|
|||
|
|
urls in the same way as the <code class="docutils literal notranslate"><span class="pre">urlopen</span></code> function: there’s no need to call
|
|||
|
|
<code class="docutils literal notranslate"><span class="pre">install_opener</span></code>, except as a convenience.</p>
|
|||
|
|
</div>
|
|||
|
|
<div class="section" id="id5">
|
|||
|
|
<h2>Basic Authentication<a class="headerlink" href="#id5" title="Permalink to this headline">¶</a></h2>
|
|||
|
|
<p>To illustrate creating and installing a handler we will use the
|
|||
|
|
<code class="docutils literal notranslate"><span class="pre">HTTPBasicAuthHandler</span></code>. For a more detailed discussion of this subject –
|
|||
|
|
including an explanation of how Basic Authentication works - see the <a class="reference external" href="http://www.voidspace.org.uk/python/articles/authentication.shtml">Basic
|
|||
|
|
Authentication Tutorial</a>.</p>
|
|||
|
|
<p>When authentication is required, the server sends a header (as well as the 401
|
|||
|
|
error code) requesting authentication. This specifies the authentication scheme
|
|||
|
|
and a ‘realm’. The header looks like: <code class="docutils literal notranslate"><span class="pre">WWW-Authenticate:</span> <span class="pre">SCHEME</span>
|
|||
|
|
<span class="pre">realm="REALM"</span></code>.</p>
|
|||
|
|
<p>e.g.</p>
|
|||
|
|
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>WWW-Authenticate: Basic realm="cPanel Users"
|
|||
|
|
</pre></div>
|
|||
|
|
</div>
|
|||
|
|
<p>The client should then retry the request with the appropriate name and password
|
|||
|
|
for the realm included as a header in the request. This is ‘basic
|
|||
|
|
authentication’. In order to simplify this process we can create an instance of
|
|||
|
|
<code class="docutils literal notranslate"><span class="pre">HTTPBasicAuthHandler</span></code> and an opener to use this handler.</p>
|
|||
|
|
<p>The <code class="docutils literal notranslate"><span class="pre">HTTPBasicAuthHandler</span></code> uses an object called a password manager to handle
|
|||
|
|
the mapping of URLs and realms to passwords and usernames. If you know what the
|
|||
|
|
realm is (from the authentication header sent by the server), then you can use a
|
|||
|
|
<code class="docutils literal notranslate"><span class="pre">HTTPPasswordMgr</span></code>. Frequently one doesn’t care what the realm is. In that
|
|||
|
|
case, it is convenient to use <code class="docutils literal notranslate"><span class="pre">HTTPPasswordMgrWithDefaultRealm</span></code>. This allows
|
|||
|
|
you to specify a default username and password for a URL. This will be supplied
|
|||
|
|
in the absence of you providing an alternative combination for a specific
|
|||
|
|
realm. We indicate this by providing <code class="docutils literal notranslate"><span class="pre">None</span></code> as the realm argument to the
|
|||
|
|
<code class="docutils literal notranslate"><span class="pre">add_password</span></code> method.</p>
|
|||
|
|
<p>The top-level URL is the first URL that requires authentication. URLs “deeper”
|
|||
|
|
than the URL you pass to .add_password() will also match.</p>
|
|||
|
|
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># create a password manager</span>
|
|||
|
|
<span class="n">password_mgr</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">HTTPPasswordMgrWithDefaultRealm</span><span class="p">()</span>
|
|||
|
|
|
|||
|
|
<span class="c1"># Add the username and password.</span>
|
|||
|
|
<span class="c1"># If we knew the realm, we could use it instead of None.</span>
|
|||
|
|
<span class="n">top_level_url</span> <span class="o">=</span> <span class="s2">"http://example.com/foo/"</span>
|
|||
|
|
<span class="n">password_mgr</span><span class="o">.</span><span class="n">add_password</span><span class="p">(</span><span class="kc">None</span><span class="p">,</span> <span class="n">top_level_url</span><span class="p">,</span> <span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span>
|
|||
|
|
|
|||
|
|
<span class="n">handler</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">HTTPBasicAuthHandler</span><span class="p">(</span><span class="n">password_mgr</span><span class="p">)</span>
|
|||
|
|
|
|||
|
|
<span class="c1"># create "opener" (OpenerDirector instance)</span>
|
|||
|
|
<span class="n">opener</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">build_opener</span><span class="p">(</span><span class="n">handler</span><span class="p">)</span>
|
|||
|
|
|
|||
|
|
<span class="c1"># use the opener to fetch a URL</span>
|
|||
|
|
<span class="n">opener</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">a_url</span><span class="p">)</span>
|
|||
|
|
|
|||
|
|
<span class="c1"># Install the opener.</span>
|
|||
|
|
<span class="c1"># Now all calls to urllib.request.urlopen use our opener.</span>
|
|||
|
|
<span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">install_opener</span><span class="p">(</span><span class="n">opener</span><span class="p">)</span>
|
|||
|
|
</pre></div>
|
|||
|
|
</div>
|
|||
|
|
<div class="admonition note">
|
|||
|
|
<p class="admonition-title">Note</p>
|
|||
|
|
<p>In the above example we only supplied our <code class="docutils literal notranslate"><span class="pre">HTTPBasicAuthHandler</span></code> to
|
|||
|
|
<code class="docutils literal notranslate"><span class="pre">build_opener</span></code>. By default openers have the handlers for normal situations
|
|||
|
|
– <code class="docutils literal notranslate"><span class="pre">ProxyHandler</span></code> (if a proxy setting such as an <span class="target" id="index-3"></span><code class="xref std std-envvar docutils literal notranslate"><span class="pre">http_proxy</span></code>
|
|||
|
|
environment variable is set), <code class="docutils literal notranslate"><span class="pre">UnknownHandler</span></code>, <code class="docutils literal notranslate"><span class="pre">HTTPHandler</span></code>,
|
|||
|
|
<code class="docutils literal notranslate"><span class="pre">HTTPDefaultErrorHandler</span></code>, <code class="docutils literal notranslate"><span class="pre">HTTPRedirectHandler</span></code>, <code class="docutils literal notranslate"><span class="pre">FTPHandler</span></code>,
|
|||
|
|
<code class="docutils literal notranslate"><span class="pre">FileHandler</span></code>, <code class="docutils literal notranslate"><span class="pre">DataHandler</span></code>, <code class="docutils literal notranslate"><span class="pre">HTTPErrorProcessor</span></code>.</p>
|
|||
|
|
</div>
|
|||
|
|
<p><code class="docutils literal notranslate"><span class="pre">top_level_url</span></code> is in fact <em>either</em> a full URL (including the ‘http:’ scheme
|
|||
|
|
component and the hostname and optionally the port number)
|
|||
|
|
e.g. <code class="docutils literal notranslate"><span class="pre">"http://example.com/"</span></code> <em>or</em> an “authority” (i.e. the hostname,
|
|||
|
|
optionally including the port number) e.g. <code class="docutils literal notranslate"><span class="pre">"example.com"</span></code> or <code class="docutils literal notranslate"><span class="pre">"example.com:8080"</span></code>
|
|||
|
|
(the latter example includes a port number). The authority, if present, must
|
|||
|
|
NOT contain the “userinfo” component - for example <code class="docutils literal notranslate"><span class="pre">"joe:password@example.com"</span></code> is
|
|||
|
|
not correct.</p>
|
|||
|
|
</div>
|
|||
|
|
<div class="section" id="proxies">
|
|||
|
|
<h2>Proxies<a class="headerlink" href="#proxies" title="Permalink to this headline">¶</a></h2>
|
|||
|
|
<p><strong>urllib</strong> will auto-detect your proxy settings and use those. This is through
|
|||
|
|
the <code class="docutils literal notranslate"><span class="pre">ProxyHandler</span></code>, which is part of the normal handler chain when a proxy
|
|||
|
|
setting is detected. Normally that’s a good thing, but there are occasions
|
|||
|
|
when it may not be helpful <a class="footnote-reference brackets" href="#id12" id="id6">5</a>. One way to do this is to setup our own
|
|||
|
|
<code class="docutils literal notranslate"><span class="pre">ProxyHandler</span></code>, with no proxies defined. This is done using similar steps to
|
|||
|
|
setting up a <a class="reference external" href="http://www.voidspace.org.uk/python/articles/authentication.shtml">Basic Authentication</a> handler:</p>
|
|||
|
|
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">proxy_support</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">ProxyHandler</span><span class="p">({})</span>
|
|||
|
|
<span class="gp">>>> </span><span class="n">opener</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">build_opener</span><span class="p">(</span><span class="n">proxy_support</span><span class="p">)</span>
|
|||
|
|
<span class="gp">>>> </span><span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">install_opener</span><span class="p">(</span><span class="n">opener</span><span class="p">)</span>
|
|||
|
|
</pre></div>
|
|||
|
|
</div>
|
|||
|
|
<div class="admonition note">
|
|||
|
|
<p class="admonition-title">Note</p>
|
|||
|
|
<p>Currently <code class="docutils literal notranslate"><span class="pre">urllib.request</span></code> <em>does not</em> support fetching of <code class="docutils literal notranslate"><span class="pre">https</span></code> locations
|
|||
|
|
through a proxy. However, this can be enabled by extending urllib.request as
|
|||
|
|
shown in the recipe <a class="footnote-reference brackets" href="#id13" id="id7">6</a>.</p>
|
|||
|
|
</div>
|
|||
|
|
<div class="admonition note">
|
|||
|
|
<p class="admonition-title">Note</p>
|
|||
|
|
<p><code class="docutils literal notranslate"><span class="pre">HTTP_PROXY</span></code> will be ignored if a variable <code class="docutils literal notranslate"><span class="pre">REQUEST_METHOD</span></code> is set; see
|
|||
|
|
the documentation on <a class="reference internal" href="../library/urllib.request.html#urllib.request.getproxies" title="urllib.request.getproxies"><code class="xref py py-func docutils literal notranslate"><span class="pre">getproxies()</span></code></a>.</p>
|
|||
|
|
</div>
|
|||
|
|
</div>
|
|||
|
|
<div class="section" id="sockets-and-layers">
|
|||
|
|
<h2>Sockets and Layers<a class="headerlink" href="#sockets-and-layers" title="Permalink to this headline">¶</a></h2>
|
|||
|
|
<p>The Python support for fetching resources from the web is layered. urllib uses
|
|||
|
|
the <a class="reference internal" href="../library/http.client.html#module-http.client" title="http.client: HTTP and HTTPS protocol client (requires sockets)."><code class="xref py py-mod docutils literal notranslate"><span class="pre">http.client</span></code></a> library, which in turn uses the socket library.</p>
|
|||
|
|
<p>As of Python 2.3 you can specify how long a socket should wait for a response
|
|||
|
|
before timing out. This can be useful in applications which have to fetch web
|
|||
|
|
pages. By default the socket module has <em>no timeout</em> and can hang. Currently,
|
|||
|
|
the socket timeout is not exposed at the http.client or urllib.request levels.
|
|||
|
|
However, you can set the default timeout globally for all sockets using</p>
|
|||
|
|
<div class="highlight-python3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">socket</span>
|
|||
|
|
<span class="kn">import</span> <span class="nn">urllib.request</span>
|
|||
|
|
|
|||
|
|
<span class="c1"># timeout in seconds</span>
|
|||
|
|
<span class="n">timeout</span> <span class="o">=</span> <span class="mi">10</span>
|
|||
|
|
<span class="n">socket</span><span class="o">.</span><span class="n">setdefaulttimeout</span><span class="p">(</span><span class="n">timeout</span><span class="p">)</span>
|
|||
|
|
|
|||
|
|
<span class="c1"># this call to urllib.request.urlopen now uses the default timeout</span>
|
|||
|
|
<span class="c1"># we have set in the socket module</span>
|
|||
|
|
<span class="n">req</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="s1">'http://www.voidspace.org.uk'</span><span class="p">)</span>
|
|||
|
|
<span class="n">response</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">req</span><span class="p">)</span>
|
|||
|
|
</pre></div>
|
|||
|
|
</div>
|
|||
|
|
</div>
|
|||
|
|
<hr class="docutils" />
|
|||
|
|
<div class="section" id="footnotes">
|
|||
|
|
<h2>Footnotes<a class="headerlink" href="#footnotes" title="Permalink to this headline">¶</a></h2>
|
|||
|
|
<p>This document was reviewed and revised by John Lee.</p>
|
|||
|
|
<dl class="footnote brackets">
|
|||
|
|
<dt class="label" id="id8"><span class="brackets"><a class="fn-backref" href="#id1">1</a></span></dt>
|
|||
|
|
<dd><p>Google for example.</p>
|
|||
|
|
</dd>
|
|||
|
|
<dt class="label" id="id9"><span class="brackets"><a class="fn-backref" href="#id2">2</a></span></dt>
|
|||
|
|
<dd><p>Browser sniffing is a very bad practice for website design - building
|
|||
|
|
sites using web standards is much more sensible. Unfortunately a lot of
|
|||
|
|
sites still send different versions to different browsers.</p>
|
|||
|
|
</dd>
|
|||
|
|
<dt class="label" id="id10"><span class="brackets"><a class="fn-backref" href="#id3">3</a></span></dt>
|
|||
|
|
<dd><p>The user agent for MSIE 6 is
|
|||
|
|
<em>‘Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)’</em></p>
|
|||
|
|
</dd>
|
|||
|
|
<dt class="label" id="id11"><span class="brackets"><a class="fn-backref" href="#id4">4</a></span></dt>
|
|||
|
|
<dd><p>For details of more HTTP request headers, see
|
|||
|
|
<a class="reference external" href="http://jkorpela.fi/http.html">Quick Reference to HTTP Headers</a>.</p>
|
|||
|
|
</dd>
|
|||
|
|
<dt class="label" id="id12"><span class="brackets"><a class="fn-backref" href="#id6">5</a></span></dt>
|
|||
|
|
<dd><p>In my case I have to use a proxy to access the internet at work. If you
|
|||
|
|
attempt to fetch <em>localhost</em> URLs through this proxy it blocks them. IE
|
|||
|
|
is set to use the proxy, which urllib picks up on. In order to test
|
|||
|
|
scripts with a localhost server, I have to prevent urllib from using
|
|||
|
|
the proxy.</p>
|
|||
|
|
</dd>
|
|||
|
|
<dt class="label" id="id13"><span class="brackets"><a class="fn-backref" href="#id7">6</a></span></dt>
|
|||
|
|
<dd><p>urllib opener for SSL proxy (CONNECT method): <a class="reference external" href="https://code.activestate.com/recipes/456195/">ASPN Cookbook Recipe</a>.</p>
|
|||
|
|
</dd>
|
|||
|
|
</dl>
|
|||
|
|
</div>
|
|||
|
|
</div>
|
|||
|
|
|
|||
|
|
|
|||
|
|
</div>
|
|||
|
|
</div>
|
|||
|
|
</div>
|
|||
|
|
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
|
|||
|
|
<div class="sphinxsidebarwrapper">
|
|||
|
|
<h3><a href="../contents.html">Table of Contents</a></h3>
|
|||
|
|
<ul>
|
|||
|
|
<li><a class="reference internal" href="#">HOWTO Fetch Internet Resources Using The urllib Package</a><ul>
|
|||
|
|
<li><a class="reference internal" href="#introduction">Introduction</a></li>
|
|||
|
|
<li><a class="reference internal" href="#fetching-urls">Fetching URLs</a><ul>
|
|||
|
|
<li><a class="reference internal" href="#data">Data</a></li>
|
|||
|
|
<li><a class="reference internal" href="#headers">Headers</a></li>
|
|||
|
|
</ul>
|
|||
|
|
</li>
|
|||
|
|
<li><a class="reference internal" href="#handling-exceptions">Handling Exceptions</a><ul>
|
|||
|
|
<li><a class="reference internal" href="#urlerror">URLError</a></li>
|
|||
|
|
<li><a class="reference internal" href="#httperror">HTTPError</a><ul>
|
|||
|
|
<li><a class="reference internal" href="#error-codes">Error Codes</a></li>
|
|||
|
|
</ul>
|
|||
|
|
</li>
|
|||
|
|
<li><a class="reference internal" href="#wrapping-it-up">Wrapping it Up</a><ul>
|
|||
|
|
<li><a class="reference internal" href="#number-1">Number 1</a></li>
|
|||
|
|
<li><a class="reference internal" href="#number-2">Number 2</a></li>
|
|||
|
|
</ul>
|
|||
|
|
</li>
|
|||
|
|
</ul>
|
|||
|
|
</li>
|
|||
|
|
<li><a class="reference internal" href="#info-and-geturl">info and geturl</a></li>
|
|||
|
|
<li><a class="reference internal" href="#openers-and-handlers">Openers and Handlers</a></li>
|
|||
|
|
<li><a class="reference internal" href="#id5">Basic Authentication</a></li>
|
|||
|
|
<li><a class="reference internal" href="#proxies">Proxies</a></li>
|
|||
|
|
<li><a class="reference internal" href="#sockets-and-layers">Sockets and Layers</a></li>
|
|||
|
|
<li><a class="reference internal" href="#footnotes">Footnotes</a></li>
|
|||
|
|
</ul>
|
|||
|
|
</li>
|
|||
|
|
</ul>
|
|||
|
|
|
|||
|
|
<h4>Previous topic</h4>
|
|||
|
|
<p class="topless"><a href="unicode.html"
|
|||
|
|
title="previous chapter">Unicode HOWTO</a></p>
|
|||
|
|
<h4>Next topic</h4>
|
|||
|
|
<p class="topless"><a href="argparse.html"
|
|||
|
|
title="next chapter">Argparse Tutorial</a></p>
|
|||
|
|
<div role="note" aria-label="source link">
|
|||
|
|
<h3>This Page</h3>
|
|||
|
|
<ul class="this-page-menu">
|
|||
|
|
<li><a href="../bugs.html">Report a Bug</a></li>
|
|||
|
|
<li>
|
|||
|
|
<a href="https://github.com/python/cpython/blob/3.7/Doc/howto/urllib2.rst"
|
|||
|
|
rel="nofollow">Show Source
|
|||
|
|
</a>
|
|||
|
|
</li>
|
|||
|
|
</ul>
|
|||
|
|
</div>
|
|||
|
|
</div>
|
|||
|
|
</div>
|
|||
|
|
<div class="clearer"></div>
|
|||
|
|
</div>
|
|||
|
|
<div class="related" role="navigation" aria-label="related navigation">
|
|||
|
|
<h3>Navigation</h3>
|
|||
|
|
<ul>
|
|||
|
|
<li class="right" style="margin-right: 10px">
|
|||
|
|
<a href="../genindex.html" title="General Index"
|
|||
|
|
>index</a></li>
|
|||
|
|
<li class="right" >
|
|||
|
|
<a href="../py-modindex.html" title="Python Module Index"
|
|||
|
|
>modules</a> |</li>
|
|||
|
|
<li class="right" >
|
|||
|
|
<a href="argparse.html" title="Argparse Tutorial"
|
|||
|
|
>next</a> |</li>
|
|||
|
|
<li class="right" >
|
|||
|
|
<a href="unicode.html" title="Unicode HOWTO"
|
|||
|
|
>previous</a> |</li>
|
|||
|
|
<li><img src="../_static/py.png" alt=""
|
|||
|
|
style="vertical-align: middle; margin-top: -1px"/></li>
|
|||
|
|
<li><a href="https://www.python.org/">Python</a> »</li>
|
|||
|
|
<li>
|
|||
|
|
<span class="language_switcher_placeholder">en</span>
|
|||
|
|
<span class="version_switcher_placeholder">3.7.4</span>
|
|||
|
|
<a href="../index.html">Documentation </a> »
|
|||
|
|
</li>
|
|||
|
|
|
|||
|
|
<li class="nav-item nav-item-1"><a href="index.html" >Python HOWTOs</a> »</li>
|
|||
|
|
<li class="right">
|
|||
|
|
|
|||
|
|
|
|||
|
|
<div class="inline-search" style="display: none" role="search">
|
|||
|
|
<form class="inline-search" action="../search.html" method="get">
|
|||
|
|
<input placeholder="Quick search" type="text" name="q" />
|
|||
|
|
<input type="submit" value="Go" />
|
|||
|
|
<input type="hidden" name="check_keywords" value="yes" />
|
|||
|
|
<input type="hidden" name="area" value="default" />
|
|||
|
|
</form>
|
|||
|
|
</div>
|
|||
|
|
<script type="text/javascript">$('.inline-search').show(0);</script>
|
|||
|
|
|
|
|||
|
|
</li>
|
|||
|
|
|
|||
|
|
</ul>
|
|||
|
|
</div>
|
|||
|
|
<div class="footer">
|
|||
|
|
© <a href="../copyright.html">Copyright</a> 2001-2019, Python Software Foundation.
|
|||
|
|
<br />
|
|||
|
|
The Python Software Foundation is a non-profit corporation.
|
|||
|
|
<a href="https://www.python.org/psf/donations/">Please donate.</a>
|
|||
|
|
<br />
|
|||
|
|
Last updated on Jul 13, 2019.
|
|||
|
|
<a href="../bugs.html">Found a bug</a>?
|
|||
|
|
<br />
|
|||
|
|
Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 2.0.1.
|
|||
|
|
</div>
|
|||
|
|
|
|||
|
|
</body>
|
|||
|
|
</html>
|