码迷,mamicode.com
首页 > 其他好文 > 详细

flink部署操作-flink standalone集群安装部署

时间:2019-03-01 17:05:25      阅读:233      评论:0      收藏:0      [点我收藏+]

标签:开始   节点   yam   data   result   集群安装   ref   export   定义   

 

flink集群安装部署

 

standalone集群模式

 

  • 必须依赖
  1. 必须的软件
  2. JAVA_HOME配置
  • flink安装
  1. 配置flink
  2. 启动flink
  3. 添加Jobmanager/taskmanager 实例到集群
  • 个人真实环境实践安装步骤

 

必须依赖

必须的软件

flink运行在所有类unix环境中,例如:linux、mac、或者cygwin,并且集群由一个master节点和一个或者多个worker节点。在你开始安装系统之前,确保你有在每个节点上安装以下软件。

 

  • java 1.8.x或者更高
  • ssh

如果你的集群没有这些软件,你需要安装或者升级他们。注意:一般linux服务器上都有ssh,但是java是需要自己安装的。

在集群的所有节点上需要配置SSH免密码登录。

JAVA_HOME配置

flink需要在集群的所有节点(master节点和worker节点)配置JAVA_HOME,指向安装在机器上的java。

你可以在这个文件中进行配置:conf/flink-conf.yaml  通过env.java.home这个key。

 

flink安装

下载页面随时下载安装包。确保选择flink安装包匹配到你的hadoop版本。如果你不打算使用hadoop的话,可以选择任意版本。

下载最新版本之后,把安装包上传到你的master节点,然后解压:

  1.  
    tar xzf flink-*.tgz
  2.  
    cd flink-*

配置flink

解压之后,需要修改conf/flink-conf.yaml

设置jobmanager.rpc.address的值为master节点的ip或者主机名。你也可以定义每个节点上允许jvm申请的最大内存,使用jobmanager.heap.mb和taskmanager.heap.mb

这两个参数的值的单位都是MB,如果有一些节点想要分配更多的内存,可以通过覆盖这个参数的默认值 FLINK_TM_HEAP

最后,你需要提供一个节点列表作为worker节点。因为,类似于HDFS配置,修改文件conf/slaves 然后在里面输入每一个worker节点的ip/hostname 。每一个worker节点将运行一个taskmanager程序。

下面的例子说明了三个节点的配置:(ip地址从10.0.0.1到10.0.0.3 对应的主机名 master worker1 worker2)并显示配置文件的内容(需要访问所有机器的相同路径)

技术图片

  1.  
    vi /path/to/flink/conf/flink-conf.yaml
  2.  
     
  3.  
    jobmanager.rpc.address: 10.0.0.1
  4.  
     
  5.  
     
  6.  
    vi /path/to/flink/conf/slaves
  7.  
     
  8.  
    10.0.0.2
  9.  
    10.0.0.3

flink目录必须在每一个worker节点的相同路劲。你可以使用一个共享的NFS目录,或者拷贝整个flink目录到每一个worker节点。

有关配置的详细信息,请参见详细的配置页面进行查看。

下面这几个参数的配置值非常重要。

 

  • Jobmanager可用内存(jobmanager.heap.mb)
  • taskmanager可用内存(taskmanager.heap.mb)
  • 每个机器可用cpu数量(taskmanager.numberOfTaskSlots)
  • 集群中的总cpu数量(parallelism.default)
  • 节点临时目录(taskmanager.tmp.dirs)

启动flink

下面的脚本将会在本机启动一个jobmanager节点,然后通过SSH连接到slaves文件中的所有worker节点,在worker节点上面启动taskmanager。现在flink启动并且运行。在本地运行的jobmanager现在将会通过配置的RPC端口接收任务。

确认你在master节点并且进入flink目录:

bin/start-cluster.sh

停止flink,需要使用stop-cluster.sh脚本

添加jobmanager或者taskmanager实例到集群

你可以通过bin/jobmanager.sh脚本和bin/taskmanager.sh脚本向一个运行中的集群添加jobmanager和taskmanager。

添加jobmanager

bin/jobmanager.sh ((start|start-foreground) cluster)|stop|stop-all

 

添加taskmanager

bin/taskmanager.sh start|start-foreground|stop|stop-all

 

 

 

 

个人真实环境实践安装步骤

以上的内容来源于官网文档翻译

下面的内容来自于本人在真实环境的一个安装步骤:

集群环境规划:三台机器,一主两从

  1.  
    hadoop100 jobManager
  2.  
    hadoop101 taskManager
  3.  
    hadoop102 taskManager
  4.  
     
  5.  
    注意:
  6.  
    1:这几台节点之间需要互相配置好SSH免密码登录。(至少要配置hadoop100可以免密码登录hadoop101和hadoop102)
  7.  
    2:这几台节点需要安装jdk1.8及以上,并且在/etc/profile中配置环境变量JAVA_HOME
  8.  
    例如:
  9.  
    export JAVA_HOME=/usr/local/jdk
  10.  
    export PATH=.:$JAVA_HOME/bin:$PATH

1:上传flink安装包到hadoop100节点的/usr/local目录下,然后解压

  1.  
    cd /usr/local
  2.  
    tar -zxvf flink-1.4.1-bin-hadoop27-scala_2.11.tgz

2:修改hadoop100节点上的flink的配置文件

  1.  
    cd /usr/local/flink-1.4.1/conf
  2.  
    vi flink-conf.yaml
  3.  
    # 修改此参数的值,改为主节点的主机名
  4.  
    jobmanager.rpc.address: hadoop100
  5.  
     
  6.  
     
  7.  
    vi slaves
  8.  
    hadoop101
  9.  
    hadoop102

3:把修改好配置文件的flink目录拷贝到其他两个节点

  1.  
    scp -rq /usr/local/flink-1.4.1 hadoop101:/usr/local
  2.  
    scp -rq /usr/local/flink-1.4.1 hadoop102:/usr/local

4:在hadoop100节点启动集群

  1.  
    cd /usr/local/flink-1.4.1
  2.  
    bin/start-cluster.sh

执行上面命令以后正常将会看到以下日志输出:

  1.  
    Using the result of ‘hadoop classpath‘ to augment the Hadoop classpath: /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
  2.  
    Starting cluster.
  3.  
    Using the result of ‘hadoop classpath‘ to augment the Hadoop classpath: /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
  4.  
    Using the result of ‘hadoop classpath‘ to augment the Hadoop classpath: /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
  5.  
    Starting jobmanager daemon on host hadoop100.
  6.  
    Using the result of ‘hadoop classpath‘ to augment the Hadoop classpath: /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
  7.  
    Using the result of ‘hadoop classpath‘ to augment the Hadoop classpath: /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
  8.  
    Starting taskmanager daemon on host hadoop101.
  9.  
    Starting taskmanager daemon on host hadoop102.

5:验证集群启动情况

查看进程:

  1.  
    在hadoop100节点上执行jps,可以看到:
  2.  
    3785 JobManager
  3.  
     
  4.  
    在hadoop101节点上执行jps,可以看到:
  5.  
    2534 TaskManager
  6.  
     
  7.  
    在hadoop101节点上执行jps,可以看到:
  8.  
    2402 TaskManager
  9.  
     
  10.  
    能看到对应的jobmanager和taskmanager进程即可。

如果启动失败了,请查看对应的日志:

  1.  
    cd /usr/local/flink-1.4.1/log
  2.  
     
  3.  
    针对jobmanager节点:
  4.  
    more flink-root-jobmanager-0-hadoop100.log
  5.  
     
  6.  
    针对taskmanager节点:
  7.  
    more flink-root-taskmanager-0-hadoop101.log
  8.  
    more flink-root-taskmanager-0-hadoop102.log
  9.  
     
  10.  
    查看此日志文件中是否有异常日志信息

6:访问集群web界面

http://hadoop100:8081

技术图片

 

7:停止集群

 

  1.  
    在hadoop100节点上执行下面命令
  2.  
    cd /usr/local/flink-1.4.1
  3.  
    bin/stop-cluster.sh

执行停止命令之后将会看到下面日志输出:

  1.  
    Using the result of ‘hadoop classpath‘ to augment the Hadoop classpath: /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
  2.  
    Using the result of ‘hadoop classpath‘ to augment the Hadoop classpath: /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
  3.  
    Using the result of ‘hadoop classpath‘ to augment the Hadoop classpath: /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
  4.  
    Stopping taskmanager daemon (pid: 3321) on host hadoop101.
  5.  
    Stopping taskmanager daemon (pid: 3088) on host hadoop102.
  6.  
    Using the result of ‘hadoop classpath‘ to augment the Hadoop classpath: /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
  7.  
    Using the result of ‘hadoop classpath‘ to augment the Hadoop classpath: /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
  8.  
    Stopping jobmanager daemon (pid: 5341) on host hadoop100.

再去对应的节点上执行jps进程发现对应的jobmanager和taskmanager进程都没有了。

 

 

flink部署操作-flink standalone集群安装部署

标签:开始   节点   yam   data   result   集群安装   ref   export   定义   

原文地址:https://www.cnblogs.com/bigben0123/p/10457378.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!