spark 安装部署

标签：spark   安装   部署   分布式   内存   

    输出类似如下说明安装成功

    首先，需要配置各个机器间的相互访问：

1、 配置ssh的自动登陆(在master机上)：

    $ ssh-keygen -t dsa -P ” -f ~/.ssh/id_dsa

    完成后会在~/.ssh/(用户目录下)生成两个文件：id_dsa 和id_dsa.pub。
    再把id_dsa.pub 追加到授权key 里面(当前并没有authorized_keys文件)：

    $ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

    完成后可以实现无密码登录本机：

    $ ssh localhost

2、把master上的id_dsa.pub 文件追加到slaves机器的authorized_keys 内( 以slaves1节点为例)：

    #拷贝master的id_dsa.pub文件(在master号机器上执行)
    $ scp id_dsa.pub redmap@192.168.1.2:/home/redmap/

    <property>

        <name>io.native.lib.available</name>

        <value>true</value>

    </property>

    <property>

        <name>fs.default.name</name>

        <value>hdfs://host:9000</value>//本机的Ip地址或者域名，端口自己设置

        <description>The name of the default file system.Either the literal string "local" or a host:port for NDFS.</description>

        <final>true</final>

    </property>

    <property>

         <name>hadoop.tmp.dir</name>

         <value>file:/home/tmp</value>

    </property>

 <property>

  <name>dfs.namenode.name.dir</name>

  <value>file:/usr/hadoop23/dfs/name</value>//本机保存name数据的目录，自定义

  <description>Determines where on the local filesystem the DFS name node should store the name table.If this is a comma-delimited list of directories,then name table is replicated in all of the directories,for redundancy.</description>

  <final>true</final>

 </property>

 <property>

  <name>dfs.datanode.data.dir</name>

  <value>file:/us/hadoop23/dfs/data</value>//本机保存data数据的目录，自定义

  <description>Determines where on the local filesystem an DFS data node should store its blocks.If this is a comma-delimited list of directories,then data will be stored in all named directories,typically on different devices.Directories that do not exist are ignored.

  </description>

  <final>true</final>

 </property>

 <property>

  <name>dfs.replication</name>

  <value>1</value>

 </property>

 <property>

  <name>dfs.permission</name>

  <value>false</value>

 </property>

 <property>

  <name>dfs.webhdfs.enabled</name>

  <value>true</value>

 </property>

file:/usr/hadoop23/dfs/name与

file:/usr/hadoop23/dfs/data

是计算机中的一些文件夹，用于存放数据和编辑文件的路径必须用一个详细的URI描述。

<property>

<name>yarn.resourcemanager.address</name>

<value>host:port</value>//本机的Ip地址或者域名，端口自己设置

<description>the host is the hostname of the ResourceManager and the port is the port on

 which the clients can talk to the Resource Manager. </description>

</property>

<property>

<name>yarn.resourcemanager.scheduler.address</name>

<value> host:port </value>//本机的Ip地址或者域名，端口自己设置

<description>host is the hostname of the resourcemanager and port is the port

    on which the Applications in the cluster talk to the Resource Manager.

 </description>

</property>

<property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value> host:port </value>//本机的Ip地址或者域名，端口自己设置

 <description>host is the hostname of the resource manager and

    port is the port on which the NodeManagers contact the Resource Manager.

 </description>

</property>

<property>

<name>yarn.resourcemanager.admin.address</name>

<value> host:8033 </value>//本机的Ip地址或者域名，端口自己设置

 <description>host is the hostname of the resource manager and

    port is the port on which the NodeManagers contact the Resource Manager.

 </description>

</property>

<property>

<name>yarn.resourcemanager.webapp.address</name>

<value> host:8088 </value>//本机的Ip地址或者域名，端口自己设置

 <description>host is the hostname of the resource manager and

    port is the port on which the NodeManagers contact the Resource Manager.

 </description>

</property>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce.shuffle</value>

</property>

<property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

8451 SecondaryNameNode

8721 NodeManager

8592 ResourceManager

9384 Jps

8152 NameNode

8282 DataNode

spark 安装部署

标签：spark   安装   部署   分布式   内存   

原文地址：http://blog.csdn.net/jyl1798/article/details/42521927