码迷,mamicode.com
首页 > 其他好文 > 详细

【原创】大数据基础之Ambari(2)通过Ambari部署ElasticSearch

时间:2019-01-17 14:00:01      阅读:769      评论:0      收藏:0      [点我收藏+]

标签:分享   .exe   过程   repeat   hid   user   rbo   org   div   

ambari的hdp中原生不支持elasticsearch安装,下面介绍如何通过mpack方式使ambari支持elasticsearch安装

安装过程

1 下载

Mpack include version 6.3.2 of ElasticSearch, Logstash, Kibana, FileBeat, and MetricBeat

# wget https://community.hortonworks.com/storage/attachments/87416-elasticsearch-mpack-2600-9.tar.gz

2 安装mpack

# ambari-server install-mpack --mpack=/path/to/87416-elasticsearch-mpack-2600-9.tar.gz --verbose

3 重启ambari-server

# ambari-server restart

访问ambari页面发现services中并没有elasticsearch,检查mpack.json发现问题:

/var/lib/ambari-server/resources/mpacks/elasticsearch-ambari.mpack-6.3.2/mpack.json

        {

          "service_name" : "ELASTICSEARCH",

          "service_version" : "6.3.2",

          "applicable_stacks" : [

            {

              "stack_name" : "HDP",

              "stack_version" : "2.3"

            },

            {

              "stack_name" : "HDP",

              "stack_version" : "2.4"

            },

            {

              "stack_name" : "HDP",

              "stack_version" : "2.5"

            },

            {

              "stack_name" : "HDP",

              "stack_version" : "2.6"

            }

          ]

        }

applicable_stacks表明service只应用到hdp2.3-2.6版本,没有3.1,手工加上;

mpack详细结构详见:https://cwiki.apache.org/confluence/display/AMBARI/Management+Packs

要注意直接修改mpack.json然后重启ambari-server不管用,需要重新install mpack:

1 卸载

ambari-server uninstall-mpack --mpack-name=elasticsearch-ambari.mpack

2 修改mpack.json并重新打包

elasticsearch_mpack-2.6.0.0-9.tar.gz

3 安装mpack(同上)

4 重启ambari-server(同上)

这时就可以在页面上看到ElasticSearch, Logstash, Kibana, FileBeat, and MetricBeat

技术分享图片

 

下面开始部署ElasticSearch,发现报错:

Traceback (most recent call last):

  File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 38, in <module>

    BeforeAnyHook().execute()

  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute

    method(env)

  File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 31, in hook

    setup_users()

  File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/shared_initialization.py", line 50, in setup_users

    groups = params.user_to_groups_dict[user],

KeyError: u‘elasticsearch‘

Error: Error: Unable to run the custom hook script [‘/usr/bin/python‘, ‘/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py‘, ‘ANY‘, ‘/var/lib/ambari-agent/data/command-303.json‘, ‘/var/lib/ambari-agent/cache/stack-hooks/before-ANY‘, ‘/var/lib/ambari-agent/data/structured-out-303.json‘, ‘INFO‘, ‘/var/lib/ambari-agent/tmp‘, ‘PROTOCOL_TLSv1_2‘, ‘‘]

跟进报错的代码:

/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/shared_initialization.py

     35   should_create_users_and_groups = False
     36   if params.host_sys_prepped:
     37     should_create_users_and_groups = not params.sysprep_skip_create_users_and_groups
     38   else:
     39     should_create_users_and_groups = not params.ignore_groupsusers_create
     40 
     41   if should_create_users_and_groups:
     42     for group in params.group_list:
     43       Group(group,
     44       )
     45 
     46     for user in params.user_list:
     47       User(user,
     48            uid = get_uid(user) if params.override_uid == "true" else None,
     49            gid = params.user_to_gid_dict[user],
     50            groups = params.user_to_groups_dict[user],
     51            fetch_nonlocal_groups = params.fetch_nonlocal_groups,
     52            )

这时需要将配置ignore_groupsusers_create打开,查看和修改配置有两种方式:

1 命令行

# cd /var/lib/ambari-server/resources/scripts
# python configs.py -u admin -p admin -n $cluster_name -l $ambari_server -t 8080 -a get -c cluster-env |grep -i ignore_groupsusers_create
"ignore_groupsusers_create": "false",
# python configs.py -u admin -p admin -n $cluster_name -l $ambari_server -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v true

详细参数详见

# python configs.py --help

2 rest api

curl --user admin:admin -i -H ‘X-Requested-By: ambari‘ -X GET "http://ambari.server:8080/api/v1/clusters/$cluster_name/configurations?type=cluster-env"

从返回的items中找到tag,然后再请求(增加tag参数):

curl --user admin:admin -i -H ‘X-Requested-By: ambari‘ -X GET "http://ambari.server:8080/api/v1/clusters/$cluster_name/configurations?type=cluster-env&tag=$tag"

返回结果中包含所有的property

"properties" : {
"agent_mounts_ignore_list" : "",
"alerts_repeat_tolerance" : "1",
"enable_external_ranger" : "false",
"fetch_nonlocal_groups" : "true",
"hide_yarn_memory_widget" : "false",
"ignore_bad_mounts" : "false",
"ignore_groupsusers_create" : "false",

修改需要先将上面的配置保存到一个文件,比如/tmp/configurations.tmp,然后修改,然后再调用接口:

curl --user admin:admin -i -H ‘X-Requested-By: ambari‘ -X PUT -d @/tmp/configurations.tmp http://ambari.server:8080/api/v1/clusters/$cluster_name

然后再安装即可;

 

参考:

https://community.hortonworks.com/articles/215078/how-to-install-elk-stack-632-in-ambari.html
https://community.hortonworks.com/questions/212871/add-custom-service-via-ambari-error.html
https://community.hortonworks.com/articles/63557/how-to-edit-the-cluster-envxml-entries-using-ambar.html

【原创】大数据基础之Ambari(2)通过Ambari部署ElasticSearch

标签:分享   .exe   过程   repeat   hid   user   rbo   org   div   

原文地址:https://www.cnblogs.com/barneywill/p/10281678.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!