标签:open flat code sel 准备 for firefox error: agent
参考:
通过测试应该是request中header的问题。
1 class S0819MtimeTiantangPipeline(object): 2 def process_item(self, item, spider): 3 headers = { 4 "User-Agent": ‘Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:53.0) Gecko/20100101 Firefox/53.0‘, 5 "Accept": ‘text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8‘, 6 "Accept-Language": ‘zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3‘, 7 "Accept-Encoding": ‘gzip, deflate‘, 8 "Upgrade-Insecure-Requests": 1, 9 ‘Connection‘: ‘keep-alive‘, 10 } 11 12 13 req = urllib2.Request(url=item[‘addr‘], headers=headers) 14 res = urllib2.urlopen(req)
下面是我怎么得到正确的header的方法:
Firefox浏览器+HttpFox插件
1. 打开HttpFox,然后将一个你要request的url输入到Firefox浏览框里,回车
例: http://img31.mtime.cn/pi/2013/01/15/163845.87188937_1000X1000.jpg
2. 如下图选取所需要的header
urllib2.HTTPError: HTTP Error 403: Forbidden 解决方法
标签:open flat code sel 准备 for firefox error: agent
原文地址:http://www.cnblogs.com/v-BigdoG-v/p/7398758.html