MAC Nutch+MySQL集成笔记

        <div property="schema:text" class="field field--name-body field--type-text-with-summary field--label-hidden field__item"><p> 目的： <em>Nutch爬虫引擎抓取的数据自动存入MySQL</em> </p>

隶属： Nutch+Hadoop+HBase(MySQL)+Elasticsearch+PHP 系列实践

MAC MySQL安装

不需要什么配置，就是next最后记住弹出的窗口里的密码就行。

下载地址： http://dev.mysql.com/downloads/mysql/

Nutch的安装与配置以及使用

1、Nutch-2.3.1下载： http://nutch.apache.org/downloads.html 下载，然后解压至本地安装目录，如本地根目录为${NUTCH_HOME}；

2、配置nutch对mysql的支持，修改${APACHE_NUTCH_HOME}/ivy/ivy.xml文件，分别：

1)找到以下行取消注释

<dependency org="mysql" name="mysql-connector-java" rev="5.1.18" conf="*->default"/>

2)修改以下行

默认为

<dependency org="org.apache.gora" name="gora-core" rev="0.3" conf="*->default"/>

修改后为

<dependency org="org.apache.gora" name="gora-core" rev="0.2.1" conf="*->default"/>

3)取消注释以下行

<dependency org="org.apache.gora" name="gora-sql" rev="0.1.1-incubating" conf="*->default" />

注释：上2)、3)如果不修改会有异常异常信息为

Exception in thread “main” Java.lang.ClassNotFoundException:org.apache.gora.sql.store.SqlStore

3、数据库连接配置

编辑${NUTCH_HOME}/conf/gora.properties文件，注释掉默认的数据库连接配置，同时添加以下配置内容：

################################ MySQL properties          ###############################gora.sqlstore.jdbc.driver=com.mysql.jdbc.Drivergora.sqlstore.jdbc.url=jdbc:mysql://192.168.58.1:3306/nutch?createDatabaseIfNotExist=truegora.sqlstore.jdbc.user=rootgora.sqlstore.jdbc.password=

写上你需要连接的数据库地址以及用户名密码

4、修改nutch-site配置文件

将以下内容添加至${NUTCH_HOME}/conf/nutch-site.xml中的configuration节点中

<property><name>http.agent.name</name><value>LiuXun Nutch Spider</value></property> <property><name>http.accept.language</name><value>ja-jp, en-us,en-gb,en;q=0.7,*;q=0.3</value><description>Value of the “Accept-Language” request header field.This allows selecting non-English language as default one to retrieve.It is a useful setting for search engines build for certain national group.</description></property> <property><name>parser.character.encoding.default</name><value>utf-8</value><description>The character encoding to fall back to when no other informationis available</description></property> <property><name>storage.data.store.class</name><value>org.apache.gora.sql.store.SqlStore</value><description>The Gora DataStore class for storing and retrieving data.Currently the following stores are available: ….</description> </property>//特别添加<property>    <name>generate.batch.id</name>    <value>*</value></property>

5、编译Nutch-2.3.1

1. 进入${NUTCH_HOME}目录下执行ant命令：ant runtime
2. 编译成功后${NUTCH_HOME}目录下会有runtime这个目录

编译Nutch

?  apache-nutch-2.3.1 (master) ? antBuildfile: /Users/hackgyj/apache-nutch-2.3.1/build.xmlTryingto overrideolddefinitionoftaskjavac  [taskdef] Couldnot loaddefinitionsfromresourceorg/sonar/ant/antlib.xml. Itcouldnot befound. ivy-probe-antlib: ivy-download:  [taskdef] Couldnot loaddefinitionsfromresourceorg/sonar/ant/antlib.xml. Itcouldnot befound. ivy-download-unchecked: ivy-init-antlib: ivy-init: init:    [mkdir] Createddir: /Users/hackgyj/apache-nutch-2.3.1/build    [mkdir] Createddir: /Users/hackgyj/apache-nutch-2.3.1/build/classes    [mkdir] Createddir: /Users/hackgyj/apache-nutch-2.3.1/build/release    [mkdir] Createddir: /Users/hackgyj/apache-nutch-2.3.1/build/test    [mkdir] Createddir: /Users/hackgyj/apache-nutch-2.3.1/build/test/classes clean-lib: resolve-default:[ivy:resolve] :: ApacheIvy 2.3.0 - 20130110142753 :: http://ant.apache.org/ivy/ ::[ivy:resolve] :: loadingsettings :: file = /Users/hackgyj/apache-nutch-2.3.1/ivy/ivysettings.xml

上面报错了，需要下载sonar的jar包（ sonar-ant-task-2.2.jar ），并将jar包放到解压好的apache-nutch-2.3.1文件夹内的lib文件内内。由于需要连接网络下载资源，需要一些时间，根据网络情况时间不等，我自己用了大概一小时！

然后命令行执行：

ant clear

再执行

ant runtime

OK，没再出错，编译成功，目录下多出：build、runtime两个文件夹，其中runtime就是编译好的目录。

6、网页抓取以及配置

1. 进入${NUTCH_HOME}/runtime/local目录下
2. 设置抓取的网站

执行命令

mkdir -p urls //建议爬虫连接文件夹echo ‘http://www.oschina.net/‘ > urls/seed.txt //写入爬取的连接bin/nutch crawl urls -depth 3 -topN 5 //开始爬虫工作

Error: JAVA_HOME is not set.

提示JAVA_HOME未设置

MAC OS X El Capitan 10.11.6 查找和设置$JAVA_HOME，命令如下

?  ~ (master) ? whichjava/usr/bin/java?  ~ (master) ? ls -l /usr/bin/javalrwxr-xr-x  1 root  wheel  74 Oct 20  2015 /usr/bin/java -> /System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java?  ~ (master) ? ls -l /System/Library/Frameworks/JavaVM.framework/Versionstotal 64lrwxr-xr-x  1 root  wheel  10 Oct 20  2015 1.4 -> CurrentJDKlrwxr-xr-x  1 root  wheel  10 Oct 20  2015 1.4.2 -> CurrentJDKlrwxr-xr-x  1 root  wheel  10 Oct 20  2015 1.5 -> CurrentJDKlrwxr-xr-x  1 root  wheel  10 Oct 20  2015 1.5.0 -> CurrentJDKlrwxr-xr-x  1 root  wheel  10 Oct 20  2015 1.6 -> CurrentJDKlrwxr-xr-x  1 root  wheel  10 Oct 20  2015 1.6.0 -> CurrentJDKdrwxr-xr-x  10 root  wheel  340 Oct  7 13:55 Alrwxr-xr-x  1 root  wheel    1 Oct 20  2015 Current -> Alrwxr-xr-x  1 root  wheel  52 Oct 20  2015 CurrentJDK -> /Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents?  ~ (master) ? java -versionjavaversion "1.8.0_91"Java(TM) SERuntimeEnvironment (build 1.8.0_91-b14)JavaHotSpot(TM) 64-BitServerVM (build 25.91-b14, mixedmode)?  ~ (master) ? /usr/libexec/java_home -VMatchingJavaVirtualMachines (3):    1.8.0_91, x86_64: "Java SE 8" /Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home    1.6.0_65-b14-468, x86_64: "Java SE 6" /Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home    1.6.0_65-b14-468, i386: "Java SE 6" /Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home /Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home//打开用户配置文件//添加路径：export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home?  ~ (master) ? open ~/.profile//保存后刷新用户配置?  ~ (master) ? source ~/.profile?  ~ (master) ? echo $JAVA_HOME/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home

Command crawl is deprecated, please use bin/crawl instead

当执行bin/nutch crawl urls -depth 3 -topN 5时显示这个错误，经查资料发现是因为Nutch2.3.1不支持这么写了。

1.7和2.2.1及以上版本用bin/crawl 取代 bin/nutch crawl .正确的写法：

bin/crawl url/ test 5

好了，能执行了，但问题又出现：

Exception in thread "main" Java.lang.NoClassDefFoundError: org/apache/avro/ipc/ByteBufferOutputStream    at java.lang.Class.forName0(Native Method)    at java.lang.Class.forName(Class.java:191)    at org.apache.nutch.storage.StorageUtils.getDataStoreClass(StorageUtils.java:93)    at org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:77)    at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:218)    at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:252)    at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:275)    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)    at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:284)Caused by: java.lang.ClassNotFoundException: org.apache.avro.ipc.ByteBufferOutputStream    at java.NET.URLClassLoader$1.run(URLClassLoader.java:366)    at java.Net.URLClassLoader$1.run(URLClassLoader.java:355)    at java.security.AccessController.doPrivileged(Native Method)    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)    ... 9 more

崩溃的感觉，再查发现答案是，Nutch2.3.1不支持MySQL，What………………

解决方法是：

1. 要么使用2.2x版本，要么退回使用nutch1.x版本
2. 或者更换MySQL为hbase存储

显示我的选择是，放弃nutch2.3.1使用nutch2.2.1。浪费我大量时间！

如出现下面的错误，请搜索本文“特别添加”来解决。

Exception in thread "main" java.lang.RuntimeException: job failed: name=generate: null, jobid=job_local200289520_0002 at org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:55) at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:199) at org.apache.nutch.crawl.Crawler.runTool(Crawler.java:68) at org.apache.nutch.crawl.Crawler.run(Crawler.java:152) at org.apache.nutch.crawl.Crawler.run(Crawler.java:250) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.nutch.crawl.Crawler.main(Crawler.java:257)

nutch2.2.1成功

?  local (master) ? bin/nutch crawl urls -depth 3 -topN 5InjectorJob: Using class org.apache.gora.sql.store.SqlStore as the Gora storage class.InjectorJob: total number of urls rejected by filters: 0InjectorJob: total number of urls injected after normalization and filtering: 1Fetcher: Your ‘http.agent.name‘ value should be listed first in ‘http.robots.agents‘ property.FetcherJob: threads: 10FetcherJob: parsing: falseFetcherJob: resuming: falseFetcherJob : timelimit set for : -1Using queue mode : byHostFetcher: threads: 10QueueFeeder finished: total 1 records. Hit by time limit :0

nutch2.2.1的安装及配置和上面一样，其中细节版本号错误等看错误信息修正就行，最后成功。

nutch命令前面章节介绍到了执行完在mysql中即查看到爬虫抓取的内容，如下图：