今天给线上服务器装爬虫环境，随便记录下安装过程，网上有很多类似的安装过程，我只是整理+验证，希望对需要安装的人有帮助

安装python

安装python

wget https://www.python.org/ftp/python/2.7.11/Python-2.7.11.tgz
tar zxvf Python-2.7.11.tgz 
cd Python-2.7.11
./configure --prefix=/usr/local
make && make altinstall

检查Python版本
python -V

安装scrapy

安装python-devel

yum search all python-devel
yum install python-devel.x86_64

安装setuptools

wget --no-check-certificate
https://bootstrap.pypa.io/ez_setup.py

–2016-05-10 10:48:35– https://bootstrap.pypa.io/ez_setup.py
Resolving bootstrap.pypa.io… 103.245.222.175
Connecting to bootstrap.pypa.io|103.245.222.175|:443… connected.
WARNING: certificate common name “*.c.ssl.fastly.net” doesn’t match requested host name “bootstrap.pypa.io”.
HTTP request sent, awaiting response… 200 OK
Length: 12385 (12K) [text/x-python]
Saving to: “ez_setup.py”

100%[==========================================================================================================>] 12,385 –.-K/s in 0s

2016-05-10 10:48:35 (105 MB/s) - “ez_setup.py” saved [12385/12385]

ll

total 83956
-rw-r–r– 1 root root 12385 May 6 21:30 ez_setup.py

python ez_setup.py --insecure

Downloading https://pypi.io/packages/source/s/setuptools/setuptools-21.0.0.zip
Extracting in /tmp/tmp9OQvdP
Now working in /tmp/tmp9OQvdP/setuptools-21.0.0
Installing Setuptools
running install
running bdist_egg
running egg_info
writing requirements to setuptools.egg-info/requires.txt
…………….
Copying setuptools-21.0.0-py2.7.egg to /usr/local/lib/python2.7/site-packages
Adding setuptools 21.0.0 to easy-install.pth file
Installing easy_install script to /usr/local/bin
Installing easy_install-2.7 script to /usr/local/bin

Installed /usr/local/lib/python2.7/site-packages/setuptools-21.0.0-py2.7.egg
Processing dependencies for setuptools==21.0.0
Finished processing dependencies for setuptools==21.0.0

pip安装

1. 下载文件
   注：如果第一次执行没有下载下来，可以多执行几次wget

wget https://bootstrap.pypa.io/get-pip.py --no-check-certificate

1. 执行安装

python get-pip.py

1. 查看版本

pip –V

zope.interface安装

网页地址：https://pypi.python.org/pypi/zope.interface/4.1.3#downloads

tar -zxvf zope.interface-4.1.3.tar.gz zope.interface-4.1.3-py2.7-linux-x86_64.egg
cd zope.interface-4.1.3
python setup.py build
python setup.py install

安装完成后，可以在/usr/local/lib/python2.7/site-packages下面看到
zope.interface-4.1.3-py2.7-linux-x86_64.egg

安装openssl

wget https://www.openssl.org/source/openssl-1.0.2h.tar.gz
tar -zxvf openssl-1.0.2h.tar.gz 
cd openssl-1.0.2h
./config
make
make test
make install

查看openssl的版本
openssl version
若还是旧版的话，执行下面的命令：

mv /usr/bin/openssl /root/
ln -s /usr/local/ssl/bin/openssl /usr/bin/openssl

再查看版本

openssl version

OpenSSL 1.0.2h 3 May 2016

pyOpenSSL安装

yum install gcc libffi-devel python-devel openssl-devel   
yum install openssl-devel python-devel python-sphinx

下载地址：
https://pypi.python.org/pypi/pyOpenSSL/0.15.1/ pyOpenSSL-0.15.1.tar.gz

linux 上如果不成功，多执行几次easy_install

easy_install https://pypi.python.org/packages/source/p/pyOpenSSL/pyOpenSSL-0.15.1.tar.gz#md5=f447644afcbd5f0a1f47350fec63a4c6

Twisted安装

wget http://twistedmatrix.com/Releases/Twisted/12.0/Twisted-12.0.0.tar.bz2
bzip2 -d Twisted-12.0.0.tar.bz2
tar -xvf Twisted-12.0.0.tar
cd Twisted-12.0.0
python setup.py install

W3lib安装

wget --no-check-certificate http://pypi.python.org/packages/source/w/w3lib/w3lib-1.0.tar.gz
wget http://pypi.python.org/packages/source/w/w3lib/w3lib-1.0.tar.gz
[root@localhost /opt/w3lib-1.0]$ tar -xvzf w3lib-1.0.tar.gz 
[root@localhost /opt/w3lib-1.0]$ cd w3lib-1.0 
[root@localhost /opt/w3lib-1.0]$ python setup.py install

安装完成后，可以在/usr/local/lib/python2.7/site-packages下面看到w3lib和w3lib-1.0-py2.7.egg-info。

安装libxml2

下载地址如下所示：
http://download.chinaunix.net/down.php?id=28497&ResourceID=6095&site=1
libxml2-2.7.4.tar.gz

cd libxml2-2.7.4
./configure
make
make install
需要python中import libxml2成功，则
cd python
python setup.py install

注：
编译libxml2时报错，./.libs/libxml2.so: undefined reference to `gzopen64’ ：
1，下载zlib-1.2.3.tar.gz,解压，configure后，vi Makefile ，然后在CFLAGS 后面添加 -fPIC 参数，编译make然后make install
2， 重新编译libxml2，
./configure –prefix=/usr/local/libxml2 –with-zlib=/usr/local/zlib/
然后vi Makefile ，然后在CFLAGS 后面添加 -fPIC 参数，编译make然后make install
ok

lxml安装

yum search all libxslt-devel
yum install libxslt-devel.x86_64
pip install lxml

安装scrapy

wget --no-check-certificate https://pypi.python.org/packages/source/S/Scrapy/Scrapy-1.0.3.tar.gz#md5=cefb981076a26112d6f9b76cb7e5ba7d
 tar -xvzf  Scrapy-1.0.3.tar.gz  
cd Scrapy-1.0.3  
python setup.py install  

安装redis

1. 安装需要的支持环境

在安装Redis之前首要先做的是安装Unix的Tcl工具，如果不安装的话后期将无法对Redis进行测试。在后期执行make test的时候返回如下错误信息：You need tcl 8.xuyao de5 or newer in order to run the Redis test，具体的流程为：
复制代码代码如下:

cd /usr/local/src
wget http://downloads.sourceforge.net/tcl/tcl8.6.3-src.tar.gz
tar -zxvf tcl8.6.3-src.tar.gz
cd ?tcl8.6.3/unix/
./configure
make
make install

1. 安装redis

安装redis的过程非常的简单，具体教程官网也有。具体如下：http://redis.io/download
复制代码代码如下:

cd /usr/local/src
$ wget http://download.redis.io/releases/redis-3.0.5.tar.gz
tar zxvf redis-3.0.5.tar.gz
cd redis-3.0.5
make
make PREFIX=/usr/local/redis install

其中PREFIX=/usr/local/redis可以省略，省略情况下redis会默认安装到/usr/local/bin目录下。
3. 测试Redis

代码如下:

cd src
make test

通过以上命令就要可以对redis进行加大的测试。

1. 查看安装版本：

redis-server --version

mysql安装

安装mysql数据库

http://www.cnblogs.com/xiaoluo501395377/archive/2013/04/07/3003278.html

安装mysql-python

pip install mysql-python
验证：
>>> import MySQLdb
>>> 

成功！