码迷,mamicode.com
首页 > 其他好文 > 详细

CentOS7 上搭建 CDH(6.3.0)

时间:2019-09-19 12:00:30      阅读:201      评论:0      收藏:0      [点我收藏+]

标签:基本配置   gpg   MIXED   targe   splay   mtu   ror   下载安装   enforce   

这里以四台节点搭建

IPHostNameOS
192.168.8.5 h5(主) CentOS 7.5
192.168.8.6 h6(从) CentOS 7.5
192.168.8.7 h7(从) CentOS 7.5
192.168.8.8 h8(从) CentOS 7.5

 

一、节点基本配置

1.yum 源,这里用阿里的(全部节点)

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
sed -i -e /mirrors.cloud.aliyuncs.com/d -e /mirrors.aliyuncs.com/d /etc/yum.repos.d/CentOS-Base.repo
yum makecache

 

2.编辑 hosts 文件(全部节点)

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/configure_network_names.html

vim /etc/hosts

192.168.8.5 h5
192.168.8.6 h6
192.168.8.7 h7
192.168.8.8 h8

 

3.关闭防火墙(全部节点)

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/install_cdh_disable_iptables.html

systemctl disable firewalld 
systemctl stop firewalld

 

4.关闭 SELinux(全部节点)

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/install_cdh_disable_selinux.html

vim /etc/selinux/config

SELINUX=disabled

# sed -i s/SELINUX=enforcing/SELINUX=disabled/g /etc/selinux/config && setenforce 0

 

5.安装 NTP 服务(全部节点,其中一台为服务器端,其余为客户端)

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/install_cdh_enable_ntp.html

这里以 h5 为 ntp 服务器,其它节点向 h5 同步时间。

具体搭建参考:https://www.cnblogs.com/jhxxb/p/10579816.html

 

6.安装 Python2.7(使用 HUE 的节点,不使用 HUE 不用安装)

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/install_python_27.html

CentOS7 自带的就是 Python2.7,不用安装

yum install python27
source /opt/rh/python27/enable
python --version

 

二、安装 Cloudera Manager 和 CDH

1.安装 CDH 的 yum 源(全部节点)

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/configure_cm_repo.html

curl -o /etc/yum.repos.d/cloudera-manager.repo https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/cloudera-manager.repo
rpm --import https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/RPM-GPG-KEY-cloudera

 

2.安装 JDK(全部节点)

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/cdh_ig_jdk_installation.html

必须安装在 /usr/java/ 目录中,这里使用 CDH 源中带的 JDK

yum install oracle-j2sdk1.8

yum 下载安装可能会很慢,可以下载后本地安装

https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/RPMS/x86_64/

yum install /home/oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm -y

 

3.安装 Cloudera Manager Server(全部节点)

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/install_cm_server.html

主节点

yum install cloudera-manager-daemons cloudera-manager-server -y

从节点(在集群部署的时候会自动安装,但下载很慢,可以先提前本地安装)

yum install cloudera-manager-daemons cloudera-manager-agent -y

同样 yum 下载安装可能会很慢,可以下载后本地安装

 

4.安装数据库,这里选用 MySQL(主节点,也可以放在其它节点)

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_mysql.html

MySQL 数据库安装参考:https://www.cnblogs.com/jhxxb/p/11156902.html

安装完成后先修改配置

vim /etc/my.cnf

my.cnf 文件

技术图片
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0

key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space.
#Replace ‘/var/lib/mysql/mysql_binary_log‘ with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES
View Code

然后再启动,新建 CDH 所需要的库:https://www.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_mysql.html#concept_dsg_3mq_bl

技术图片

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

-- 也可以直接用 root 用户
GRANT ALL ON scm.* TO cdh6@% IDENTIFIED BY 123456;
GRANT ALL ON amon.* TO cdh6@% IDENTIFIED BY 123456;
GRANT ALL ON rman.* TO cdh6@% IDENTIFIED BY 123456;
GRANT ALL ON hue.* TO cdh6@% IDENTIFIED BY 123456;
GRANT ALL ON metastore.* TO cdh6@% IDENTIFIED BY 123456;
GRANT ALL ON sentry.* TO cdh6@% IDENTIFIED BY 123456;
GRANT ALL ON nav.* TO cdh6@% IDENTIFIED BY 123456;
GRANT ALL ON navms.* TO cdh6@% IDENTIFIED BY 123456;
GRANT ALL ON oozie.* TO cdh6@% IDENTIFIED BY 123456;

最后安装 MySQL 驱动

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_mysql.html#cmig_topic_5_5_3

yum install wget -y
# 下载
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
# 解压
tar -zxf mysql-connector-java-5.1.46.tar.gz
# 放到指定目录
mkdir -p /usr/share/java/
cp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

# 删除
rm -rf  mysql-connector-java-5.1.46 mysql-connector-java-5.1.46.tar.gz

 

5.设置 Cloudera Manager 数据库(主节点)

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/prepare_cm_database.html

格式:/opt/cloudera/cm/schema/scm_prepare_database.sh <databaseType>  <databaseName>  <databaseUser>

当 MySQL 或 MariaDB 与 Cloudera Manager Server 位于同一节点时:/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm

当 MySQL 或 MariaDB 在其它节点上时:/opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h db01.example.com --scm-host cm01.example.com scm scm

 

6.安装 CDH 和其他软件(主节点)

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/install_software_cm_wizard.html

systemctl start cloudera-scm-server
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

# 看到如下日志就可以访问 WEB 端了:http://<server_host>:7180,用户名和密码默认都是 admin
INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

子节点启动

systemctl start cloudera-scm-agent

 


http://wangshubing.com/2019/01/24/CentOS7.5,CDH6%E5%AE%89%E8%A3%85%E9%83%A8%E7%BD%B2/

https://mp.weixin.qq.com/s/c7c0wOHb7wPr6Kbg3MTU1A

CentOS7 上搭建 CDH(6.3.0)

标签:基本配置   gpg   MIXED   targe   splay   mtu   ror   下载安装   enforce   

原文地址:https://www.cnblogs.com/jhxxb/p/11515734.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!