-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathurl_fetch.html
More file actions
150 lines (136 loc) · 10.2 KB
/
Copy pathurl_fetch.html
File metadata and controls
150 lines (136 loc) · 10.2 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
<!DOCTYPE HTML>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>URL Fetch - LIZZIE</title>
<meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=0"/>
<link rel="stylesheet" href="/assets/style.css?v=9cec8"/>
<link rel="stylesheet" href="/assets/pygments.css?v=cedbb"/>
<link rel="alternate" type="application/rss+xml" href="/feed.xml" title="LIZZIE"/>
<link rel="canonical" href="/2008/url_fetch.html" />
<script>
(function() {
var cx = '012347804910294994683:rsexabosj44';
var gcse = document.createElement('script'); gcse.type = 'text/javascript'; gcse.async = true;
gcse.src = (document.location.protocol == 'https:' ? 'https:' : 'http:') +
'//www.google.com/cse/cse.js?cx=' + cx;
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(gcse, s);
})();
</script>
</head>
<body>
<!--http://vimeo.com/53043267-->
<div id="topbar"></div>
<header>
<div id="logo"><a href="/">LIZZIE</a></div>
<nav id="nav" role="navigation">
<a href="/">Home</a>
<a href="/archive.html">Archive</a>
</nav>
</header>
<div id="search">
<gcse:searchbox enableHistory="true" autoCompleteMaxCompletions="5" autoCompleteMatchType='any'></gcse:searchbox>
</div>
<div id="search-results">
<gcse:searchresults refinementStyle="link" adclient="pub-3716363443644786"></gcse:searchresults>
</div>
<div class="container"><h1 class="entry-title">URL Fetch</h1>
<div class="entry-meta">
<time class="updated" datetime="2008-11-11T14:11:39+08:00">
<a href="/2008/">2008-11-11</a>
</time>
<span class="author vcard">
by <a class="url fn" href="https://lizzie.github.io">Yan Sheng</a>
</span>
</div>
<div class="entry-content">
<p>URLFetch使用示例</p>
<p>一般使用就是直接使用urlfetch.fetch(url), 返回一个ResponseObject,其中有content, status_code等,基本使用请看对应文档 <a class="reference external" href="http://code.google.com/appengine/docs/urlfetch/">http://code.google.com/appengine/docs/urlfetch/</a>
其中想要fetch需要登录的页面,似乎不是那么简单.依次尝试以下:</p>
<ol class="arabic simple">
<li>使用curl</li>
</ol>
<p>使用现成的curl模块, 参考 <a class="reference external" href="http://curl.haxx.se/docs/httpscripting">http://curl.haxx.se/docs/httpscripting</a>.html还未尝试.</p>
<ol class="arabic simple" start="2">
<li>添加cookie进行自动登录</li>
</ol>
<p>网上一段示例代码片段</p>
<div class="highlight"><pre><span class="n">cj</span> <span class="o">=</span> <span class="n">cookielib</span><span class="o">.</span><span class="n">CookieJar</span><span class="p">()</span>
<span class="n">opener</span> <span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">build_opener</span><span class="p">(</span><span class="n">urllib2</span><span class="o">.</span><span class="n">HTTPCookieProcessor</span><span class="p">(</span><span class="n">cj</span><span class="p">))</span>
<span class="n">exheaders</span> <span class="o">=</span> <span class="p">[(</span><span class="s">"User-Agent"</span><span class="p">,</span><span class="s">"Mozilla/4.0 (compatible; MSIE 7.1; Windows NT 5.1; SV1)"</span><span class="p">),]</span>
<span class="n">opener</span><span class="o">.</span><span class="n">addheaders</span><span class="o">=</span><span class="n">exheaders</span>
<span class="n">url_login</span> <span class="o">=</span> <span class="s">'http://xiaonei.com/Login.do'</span>
<span class="n">body</span> <span class="o">=</span> <span class="p">((</span><span class="s">'email'</span><span class="p">,</span><span class="s">'xxxxx@gmail.com'</span><span class="p">),</span> <span class="p">(</span><span class="s">'password'</span><span class="p">,</span><span class="s">'*********'</span><span class="p">))</span> <span class="c">#TODO:更改登录名和密码</span>
<span class="n">req1</span> <span class="o">=</span> <span class="n">opener</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">url_login</span><span class="p">,</span> <span class="n">urllib</span><span class="o">.</span><span class="n">urlencode</span><span class="p">(</span><span class="n">body</span><span class="p">))</span> <span class="c">#这时，cookie已经进来了。</span>
</pre></div>
<p>可以看到,直接将用户名和密码放在请求头中了.</p>
<ol class="arabic simple" start="3">
<li>另外一个设置cookie实现登录</li>
</ol>
<div class="highlight"><pre><span class="c">#!/usr/bin/env python</span>
<span class="c">#coding=utf-8</span>
<span class="kn">import</span> <span class="nn">urllib2</span><span class="o">,</span><span class="nn">cookielib</span>
<span class="kn">import</span> <span class="nn">urllib</span>
<span class="kn">from</span> <span class="nn">BeautifulSoup</span> <span class="kn">import</span> <span class="n">BeautifulSoup</span>
<span class="n">cookie</span><span class="o">=</span><span class="n">cookielib</span><span class="o">.</span><span class="n">CookieJar</span><span class="p">()</span>
<span class="n">opener</span><span class="o">=</span><span class="n">urllib2</span><span class="o">.</span><span class="n">build_opener</span><span class="p">(</span>
<span class="n">urllib2</span><span class="o">.</span><span class="n">HTTPCookieProcessor</span><span class="p">(</span><span class="n">cookie</span><span class="p">))</span>
<span class="n">urllib2</span><span class="o">.</span><span class="n">install_opener</span><span class="p">(</span><span class="n">opener</span><span class="p">)</span>
<span class="n">postdata</span><span class="o">=</span><span class="n">urllib</span><span class="o">.</span><span class="n">urlencode</span><span class="p">({</span><span class="s">'username'</span><span class="p">:</span><span class="s">'python-</span>
<span class="n">cn</span><span class="s">','</span><span class="n">password</span><span class="s">':'</span><span class="n">aaaaaa</span><span class="s">'})</span>
<span class="n">login_response</span><span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="s">'http://www.meizu.com/bbs/</span>
<span class="n">login</span><span class="o">.</span><span class="n">asp</span><span class="s">' ,postdata)</span>
<span class="n">ww</span><span class="o">=</span><span class="nb">open</span><span class="p">(</span><span class="s">'f.html'</span><span class="p">,</span> <span class="s">'w'</span><span class="p">)</span>
<span class="n">url_us</span><span class="o">=</span><span class="s">r"http://www.meizu.com/bbs/infolist.asp?t=toplist&orders=7"</span>
<span class="n">page</span><span class="o">=</span><span class="n">urllib2</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">url_us</span><span class="p">)</span>
<span class="n">soup</span><span class="o">=</span><span class="n">BeautifulSoup</span><span class="p">(</span><span class="n">page</span><span class="p">)</span>
<span class="n">ww</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">soup</span><span class="p">))</span>
<span class="n">ww</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</pre></div>
<p>还是将帐号密码放在代码里了....</p>
<p>以上都是直接增加cookies,,,而想要的认证方式,后来发现是例如这样的.</p>
<ul class="simple">
<li>google帐号授权 <a class="reference external" href="http://code.google.com/apis/accounts/docs/AuthSub.html">http://code.google.com/apis/accounts/docs/AuthSub.html</a></li>
<li><a class="reference external" href="http://code.google.com/apis/gdata/auth.html#AuthSub">http://code.google.com/apis/gdata/auth.html#AuthSub</a></li>
<li><a class="reference external" href="http://code.google.com/intl/zh-CN/appengine/articles/gdata.html">http://code.google.com/intl/zh-CN/appengine/articles/gdata.html</a></li>
<li><a class="reference external" href="http://code.google.com/appengine/docs/usinggdataservices.html">http://code.google.com/appengine/docs/usinggdataservices.html</a></li>
<li><a class="reference external" href="http://code.google.com/apis/gdata/articles/python_client_lib.html">http://code.google.com/apis/gdata/articles/python_client_lib.html</a></li>
<li><a class="reference external" href="http://code.google.com/apis/gdata/authsub.html">http://code.google.com/apis/gdata/authsub.html</a></li>
<li><a class="reference external" href="http://code.google.com/apis/gdata/">http://code.google.com/apis/gdata/</a></li>
</ul>
<p>,,,还没弄明白.待续...</p>
</div>
<div class="entry-tags">
<a href="/tag/#GAE">GAE</a><a href="/tag/#Python">Python</a>
</div>
<div id="disqus_thread"></div>
<script type="text/javascript">
var disqus_shortname = 'sunsetsunrising';
var disqus_title = 'URL Fetch';
(function() {
var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
})();
</script>
</div>
<footer>
<hr class="end"/>
<p>©2008~2012 <a href="https://github.com/lizzie/lizzie.github.com">Liz</a> ♥ <span class="software">Powered by <a href="http://lab.lepture.com/liquidluck/">Felix Felicis 3.5</a></span></p>
</footer>
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-4419044-5']);
_gaq.push(['_trackPageview']);
_gaq.push(['_trackPageLoadTime']);
(function () {
var ga = document.createElement('script');
ga.type = 'text/javascript';
ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0];
s.parentNode.insertBefore(ga, s);
})();
</script>
</body>
</html>