码迷,mamicode.com
首页 > 其他好文 > 详细

初识 scrapy 框架 - 安装

时间:2018-01-25 10:58:50      阅读:213      评论:0      收藏:0      [点我收藏+]

标签:ping   http   value   ssi   6.2   using   lib   ssl   comm   

前面豆子学习了基本的urllib的模块,通过这个模块可以写一些简单的爬虫文件。如果要处理大中型的爬虫项目,urllib就显得比较low了,这个时候可以使用scrapy框架来实现,很多基本的处理在scrapy里面已经做好了。

首先来安装一下。推荐的流程如下:

  1. 首先升级一下pip
C:\WINDOWS\system32>python -m pip install --upgrade pip
Requirement already up-to-date: pip in c:\python36\lib\site-packages
  1. 安装wheel
    C:\WINDOWS\system32>pip install wheel
    Requirement already satisfied: wheel in c:\python36\lib\site-packages

3.安装lxml

C:\WINDOWS\system32>pip install lxml
Collecting lxml
  Downloading lxml-4.1.1-cp36-cp36m-win32.whl (3.2MB)
    100% |████████████████████████████████| 3.2MB 307kB/s
Installing collected packages: lxml
Successfully installed lxml-4.1.1
  1. 安装twisted(如果在线安装报错的话,可以考虑离线安装)
    从https://www.lfd.uci.edu 上搜索twisted, 可以看见他有很多个版本

技术分享图片

执行一下python命令看看当前的版本,可以看见我的版本是3.6.2,32位的

c:\Users\yuan.li\Downloads>python
Python 3.6.2 (v3.6.2:5fd33b5, Jul  8 2017, 04:14:34) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

这样的话可以下载对应的版本twisted-17.9.0-cp36-win32.whl就行了。
下载之后,手动安装

c:\Users\yuan.li\Downloads>pip install Twisted-17.9.0-cp36-cp36m-win32.whl
Processing c:\users\yuan.li\downloads\twisted-17.9.0-cp36-cp36m-win32.whl
Requirement already satisfied: incremental>=16.10.1 in c:\python36\lib\site-packages (from Twisted==17.9.0)
Requirement already satisfied: Automat>=0.3.0 in c:\python36\lib\site-packages (from Twisted==17.9.0)
Requirement already satisfied: zope.interface>=4.0.2 in c:\python36\lib\site-packages (from Twisted==17.9.0)
Requirement already satisfied: hyperlink>=17.1.1 in c:\python36\lib\site-packages (from Twisted==17.9.0)
Requirement already satisfied: constantly>=15.1 in c:\python36\lib\site-packages (from Twisted==17.9.0)
Requirement already satisfied: six in c:\python36\lib\site-packages (from Automat>=0.3.0->Twisted==17.9.0)
Requirement already satisfied: attrs in c:\python36\lib\site-packages (from Automat>=0.3.0->Twisted==17.9.0)
Requirement already satisfied: setuptools in c:\python36\lib\site-packages (from zope.interface>=4.0.2->Twisted==17.9.0)
Installing collected packages: Twisted
Successfully installed Twisted-17.9.0
  1. 最后安装scrapy
c:\Users\yuan.li\Downloads>pip install scrapy
Collecting scrapy
  Downloading Scrapy-1.5.0-py2.py3-none-any.whl (251kB)
    100% |████████████████████████████████| 256kB 2.3MB/s
Collecting pyOpenSSL (from scrapy)
  Downloading pyOpenSSL-17.5.0-py2.py3-none-any.whl (53kB)
    100% |████████████████████████████████| 61kB 4.5MB/s
Collecting cssselect>=0.9 (from scrapy)
  Downloading cssselect-1.0.3-py2.py3-none-any.whl
Collecting parsel>=1.1 (from scrapy)

安装完成之后,执行一下scrapy, 看看是否工作

c:\Users\yuan.li\Downloads>scrapy
Scrapy 1.5.0 - no active project

Usage:
  scrapy <command> [options] [args]

Available commands:
  bench         Run quick benchmark test
  fetch         Fetch a URL using the Scrapy downloader
  genspider     Generate new spider using pre-defined templates
  runspider     Run a self-contained spider (without creating a project)
  settings      Get settings values
  shell         Interactive scraping console
  startproject  Create new project
  version       Print Scrapy version
  view          Open URL in browser, as seen by Scrapy

  [ more ]      More commands available when run from project directory

初识 scrapy 框架 - 安装

标签:ping   http   value   ssi   6.2   using   lib   ssl   comm   

原文地址:http://blog.51cto.com/beanxyz/2064882

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!