标签：cti 用户 man mysqld inactive 应用 char glib systemd
用了一周多的时间终于把CDH版Hadoop部署在了测试环境，本文将就这个部署过程做个总结。
一、Hadoop版本选择。
Hadoop大致可分为Apache Hadoop和第三方发行第三方发行版Hadoop，考虑到Hadoop集群部署的高效，集群的稳定性，以及后期集中的配置管理，业界多使用Cloudera公司的发行版，简称为CDH。
下面是转载的Hadoop社区版本与第三方发行版本的比较：
Apache社区版本
优点：
缺点：
第三方发行版本（如CDH，HDP，MapR等）
优点：
缺点：
转自：http://itindex.net/detail/51484-%E8%87%AA%E5%AD%A6-%E5%A4%A7%E6%95%B0%E6%8D%AE-%E7%94%9F%E4%BA%A7
更多内容请看原作者博客。
二、安装介质准备
安装介质准备和安装部分主要参考：http://blog.csdn.net/shawnhu007/article/details/52579204，对其内容进行少许补充以做到能傻瓜安装的目的。
我们采用离线安装的方式，需要下载CDH离线安装包和相关组件：
介质下载和安装部分主要参考：http://blog.csdn.net/shawnhu007/article/details/52579204
在线安装请参考文章（对网速有较高要求）：http://www.cnblogs.com/ee900222/p/hadoop_3.html
三、操作系统准备
准备好三台环境一样的centos7在本地虚拟机VMWare上，Cloudera发行版比起Apache社区版本安装对硬件的要求更高，内存至少10G，不然后面你会遇到各种问题，或许都找不到答案。
本人前2次安装失败就是因为节点分配内存太少，建议对于cloudera-scm-server就需要至少4G的内存，cloudera-scm-agent的内存至少也需要1.5G以上。
3台虚拟机环境如下：
| IP地址 | 主机名 | 说明 |
| 192.168.42.128 | CDH1 | 主节点master,datanode |
| 192.168.42.129 | CDH2 | datanode |
| 192.168.42.30 | CDH3 | datanode |
四、开始安装前配置和预装软件
可以在VM中先安装1台机器，做完相关配置后再克隆出另外2台机器，以避免在3台机器上的重复配置
[root@cdh1~]$ java -version [root@cdh1~]$ rpm -qa | grep jdk java-1.7.0-openjdk-1.7.0.75-2.5.4.2.el7_0.x86_64 java-1.7.0-openjdk-headless-1.7.0.75-2.5.4.2.el7_0.x86_64 [root@cdh1~]# yum -y remove java-1.7.0-openjdk-1.7.0.75-2.5.4.2.el7_0.x86_64 [root@cdh1~]# yum -y remove java-1.7.0-openjdk-headless-1.7.0.75-2.5.4.2.el7_0.x86_64 [root@cdh1~]# java -version bash: /usr/bin/java: No such file or directory [root@cdh1~]# rpm -ivh jdk-8u101-linux-x64.rpm [root@cdh1~]# java -version java version "1.8.0_101" Java(TM) SE Runtime Environment (build 1.8.0_101-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
[root@cdh1~]# vi /etc/sysconfig/network NETWORKING=yes HOSTNAME=cdh1 [root@cdh1~]# vi /etc/hosts 127.0.0.1 localhost.cdh1 192.168.42.128 cdh1 192.168.42.129 cdh2 192.168.42.130 cdh3
[root@cdh1~]# vi /etc/sysconfig/selinux SELINUX=disabled [root@cdh1~]#sestatus -v SELinux status: disabled 表示已经关闭了
[root@cdh1~]# systemctl stop firewalld [root@cdh1~]# systemctl disable firewalld rm ‘/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service‘ rm ‘/etc/systemd/system/basic.target.wants/firewalld.service‘ [root@cdh1~]# systemctl status firewalld firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled) Active: inactive (dead)
[root@cdh1~]#yum -y install ntp 更改master的节点 [root@cdh1~]## vi /etc/ntp.conf 注释掉所有server *.*.*的指向，新添加一条可连接的ntp服务器(我选的本公司的ntp测试服务器) server 172.30.0.19 iburst 在其他节点上把ntp指向master服务器地址即可(/etc/ntp.conf下) server 192.168.42.128 iburst [root@cdh1~]## systemctl start ntpd //启动ntp服务 [root@cdh1~]## systemctl status ntpd //查看ntp服务状态
下面以192.168.42.128到192.168.42.129的免密登录设置举例
[root@cdh1 /]# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): /root/.ssh/id_rsa already exists. Overwrite (y/n)? y Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: 1d:e9:b4:ed:1d:e5:c6:a7:f3:23:ac:02:2b:8c:fc:ca root@cdh1 The key‘s randomart image is: +--[ RSA 2048]----+ | | | . | | + .| | + + + | | S + . . =| | . . . +.| | . o o o + | | .o o . . o + | | Eo.. ... . o| +-----------------+ [root@cdh1 /]# ssh-copy-id 192.168.42.129 /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@192.168.42.129‘s password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh ‘192.168.42.129‘" and check to make sure that only the key(s) you wanted were added.
安装mysql
centos7自带的是mariadb，需要先卸载掉
[root@cdh1 /]# rpm -qa | grep mariadb mariadb-libs-5.5.41-2.el7_0.x86_64 [root@cdh1 /]# rpm -e --nodeps mariadb-libs-5.5.41-2.el7_0.x86_64 [root@cdh1 /]# tar -xvf MySQL-5.6.24-1.linux_glibc2.5.x86_64.rpm-bundle.tar //mysql rpm包拷贝到服务器上然后解压 [root@cdh1 /]# rpm -ivh MySQL-*.rpm //安装释出的全部rpm [root@cdh1 /]# cp /usr/share/mysql/my-default.cnf /etc/my.cnf [root@cdh1 /]# vi /etc/my.cnf //在配置文件中增加以下配置并保存 [mysqld] default-storage-engine = innodb innodb_file_per_table collation-server = utf8_general_ci init-connect = ‘SET NAMES utf8‘ character-set-server = utf8 [root@cdh1 /]# yum install -y perl-Module-Install.noarch [root@cdh1 /]# /usr/bin/mysql_install_db //初始化mysql [root@cdh1 /]# service mysql restart //启动mysql ERROR! MySQL server PID file could not be found! Starting MySQL... SUCCESS! [root@cdh1 /]# cat /root/.mysql_secret //查看mysql root初始化密码 # The random password set for the root user at Fri Sep 22 11:13:25 2017 (local time): 9mp7uYFmgt6drdq3 [root@cdh1 /]# mysql -u root -p //登录进行去更改密码 mysql> SET PASSWORD=PASSWORD(‘123456‘); mysql> update user set host=‘%‘ where user=‘root‘ and host=‘localhost‘; //允许mysql远程访问 Query OK, 1 row affected (0.05 sec) Rows matched: 1 Changed: 1 Warnings: 0 mysql> flush privileges; Query OK, 0 rows affected (0.00 sec) [root@cdh1 /]# chkconfig mysql on //配置开机启动
[root@cdh1 /]# tar -zcvf mysql-connector-java-5.1.44.tar.gz // 解压mysql-connector-java-5.1.44.tar.gz得到mysql-connector-java-5.1.44-bin.jar
[root@cdh1 /]# mkdir /usr/share/java // 在各节点创建java文件夹
[root@cdh1 /]# cp mysql-connector-java-5.1.44-bin.jar /usr/share/java/mysql-connector-java.jar //将mysql-connector-java-5.1.44-bin.jar拷贝到/usr/share/java路径下并重命名为mysql-connector-java.jar
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci; Query OK, 1 row affected (0.00 sec) create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci; Query OK, 1 row affected (0.00 sec) create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci; Query OK, 1 row affected (0.00 sec) create database monitor DEFAULT CHARSET utf8 COLLATE utf8_general_ci; Query OK, 1 row affected (0.00 sec) create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci; Query OK, 1 row affected (0.00 sec) grant all on *.* to root@"%" Identified by "123456";
五、安装Cloudera-Manager
//解压cm tar包到指定目录所有服务器都要(或者在主节点解压好，然后通过scp到各个节点同一目录下) [root@cdh1 ~]#mkdir /opt/cloudera-manager [root@cdh1 ~]# tar -axvf cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz -C /opt/cloudera-manager //创建cloudera-scm用户（所有节点） [root@cdh1 ~]# useradd --system --home=/opt/cloudera-manager/cm-5.7.2/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm //在主节点创建cloudera-manager-server的本地元数据保存目录 [root@cdh1 ~]# mkdir /var/cloudera-scm-server [root@cdh1 ~]# chown cloudera-scm:cloudera-scm /var/cloudera-scm-server [root@cdh1 ~]# chown cloudera-scm:cloudera-scm /opt/cloudera-manager //配置从节点cloudera-manger-agent指向主节点服务器 [root@cdh1 ~]# vi /opt/cloudera-manager/cm-5.7.2/etc/cloudera-scm-agent/config.ini 将server_host改为CMS所在的主机名即cdh1 //主节点中创建parcel-repo仓库目录 [root@cdh1 ~]# mkdir -p /opt/cloudera/parcel-repo [root@cdh1 ~]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo [root@cdh1 ~]# cp CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel.sha manifest.json /opt/cloudera/parcel-repo 注意：其中CDH-5.7.2-1.cdh5.7.2.p0.18-el5.parcel.sha1 后缀要把1去掉 //所有节点创建parcels目录 [root@cdh1 ~]# mkdir -p /opt/cloudera/parcels [root@cdh1 ~]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcels 解释：Clouder-Manager将CDHs从主节点的/opt/cloudera/parcel-repo目录中抽取出来，分发解压激活到各个节点的/opt/cloudera/parcels目录中 //初始脚本配置数据库scm_prepare_database.sh(在主节点上) [root@cdh1 ~]# /opt/cloudera-manager/cm-5.7.2/share/cmf/schema/scm_prepare_database.sh mysql -hcdh1 -uroot -p123456 --scm-host cdh1 scmdbn scmdbu scmdbp 说明：这个脚本就是用来创建和配置CMS需要的数据库的脚本。各参数是指： mysql：数据库用的是mysql，如果安装过程中用的oracle，那么该参数就应该改为oracle。 -cdh1：数据库建立在cdh1主机上面,也就是主节点上面。 -uroot：root身份运行mysql。-123456：mysql的root密码是***。 --scm-host cdh1：CMS的主机，一般是和mysql安装的主机是在同一个主机上,最后三个参数是：数据库名，数据库用户名，数据库密码。 如果报错： ERROR com.cloudera.enterprise.dbutil.DbProvisioner - Exception when creating/dropping database with user ‘root‘ and jdbc url ‘jdbc:mysql://localhost/?useUnicode=true&characterEncoding=UTF-8‘ java.sql.SQLException: Access denied for user ‘root‘@‘cdh1‘ (using password: YES) 则参考 http://forum.spring.io/forum/spring-projects/web/57254-java-sql-sqlexception-access-denied-for-user-root-localhost-using-password-yes 运行如下命令： update user set PASSWORD=PASSWORD(‘123456‘) where user=‘root‘; GRANT ALL PRIVILEGES ON *.* TO ‘root‘@‘cdh1‘ IDENTIFIED BY ‘123456‘ WITH GRANT OPTION; FLUSH PRIVILEGES; //启动主节点 [root@cdh1 ~]# cp /opt/cloudera-manager/cm-5.7.2/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server [root@cdh1 ~]# chkconfig cloudera-scm-server on [root@cdh1 ~]# vi /etc/init.d/cloudera-scm-server CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}改为=/opt/cloudera-manager/cm-5.7.2/etc/default [root@cdh1 ~]# service cloudera-scm-server start //同时为了保证在每次服务器重启的时候都能启动cloudera-scm-server，应该在开机启动脚本/etc/rc.local中加入命令：service cloudera-scm-server restart //启动cloudera-scm-agent所有节点 [root@cdhX ~]# mkdir /opt/cloudera-manager/cm-5.7.2/run/cloudera-scm-agent [root@cdhX ~]# cp /opt/cloudera-manager/cm-5.7.2/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent [root@cdhX ~]# chkconfig cloudera-scm-agent on [root@cdhX ~]# vi /etc/init.d/cloudera-scm-agent CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}改为=/opt/cloudera-manager/cm-5.7.2/etc/default [root@cdhX ~]# service cloudera-scm-agent start //同时为了保证在每次服务器重启的时候都能启动cloudera-scm-agent，应该在开机启动脚本/etc/rc.local中加入命令：service cloudera-scm-agent restart
标签：cti 用户 man mysqld inactive 应用 char glib systemd
原文地址：http://www.cnblogs.com/zhangleisanshi/p/7575579.html
