标签：mat url engine 应用市场 重点 mic 好的 OLE 测试域名
time: 2020-1-3
本文基于阿里云ACK托管K8S集群
分为以下几个部分:
使用Aliyun 提供的CloudShell进行操作的时候,一来默认不会保存文件,二来容易连接超时,导致安装spark operator失败,重新安装需要手动删除spark operator的各类资源.
安装Helm的方式:
mkdir -pv helm && cd helm
wget https://storage.googleapis.com/kubernetes-helm/helm-v2.9.1-linux-amd64.tar.gz
tar xf helm-v2.9.1-linux-amd64.tar.gz
sudo mv linux-amd64/helm /usr/local/bin
rm -rf linux-amd64
# 查看版本，不显示出server版本，因为还没有安装server
helm versionhelm install incubator/sparkoperator --namespace spark-operator --set sparkJobNamespace=default --set operatorImageName=registry-vpc.us-east-1.aliyuncs.com/eci_open/spark-operator --set operatorVersion=v1beta2-1.0.1-2.4.4 --set enableWebhook=true --set ingressUrlFormat="\{\{\$appName\}\}.ACK测试域名" --set enableBatchScheduler=true Note:
registry-vpc表示使用内网访问registry下载镜像.以下创建default:spark servicecount并绑定相关权限:
创建spark-rbac.yaml,并执行kubectl apply -f spark-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: spark
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: default
name: spark-role
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["*"]
- apiGroups: [""]
resources: ["services"]
verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: spark-role-binding
namespace: default
subjects:
- kind: ServiceAccount
name: spark
namespace: default
roleRef:
kind: Role
name: spark-role
apiGroup: rbac.authorization.k8s.io这里分为以下几步:
参照链接:https://help.aliyun.com/document_detail/146237.html?spm=a2c4g.11186623.2.16.4dce2e14IGuHEv
以下可以直接操作,下载到oss依赖的jar包
wget http://gosspublic.alicdn.com/hadoop-spark/hadoop-oss-hdp-2.6.1.0-129.tar.gz?spm=a2c4g.11186623.2.11.54b56c18VGGAzb&file=hadoop-oss-hdp-2.6.1.0-129.tar.gz
tar -xvf hadoop-oss-hdp-2.6.1.0-129.tar
hadoop-oss-hdp-2.6.1.0-129/
hadoop-oss-hdp-2.6.1.0-129/aliyun-java-sdk-ram-3.0.0.jar
hadoop-oss-hdp-2.6.1.0-129/aliyun-java-sdk-core-3.4.0.jar
hadoop-oss-hdp-2.6.1.0-129/aliyun-java-sdk-ecs-4.2.0.jar
hadoop-oss-hdp-2.6.1.0-129/aliyun-java-sdk-sts-3.0.0.jar
hadoop-oss-hdp-2.6.1.0-129/jdom-1.1.jar
hadoop-oss-hdp-2.6.1.0-129/aliyun-sdk-oss-3.4.1.jar
hadoop-oss-hdp-2.6.1.0-129/hadoop-aliyun-2.7.3.2.6.1.0-129.jar<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<!-- OSS配置 -->
<property>
<name>fs.oss.impl</name>
<value>org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem</value>
</property>
<property>
<name>fs.oss.endpoint</name>
<value>oss-cn-hangzhou-internal.aliyuncs.com</value>
</property>
<property>
<name>fs.oss.accessKeyId</name>
<value>{临时AK_ID}</value>
</property>
<property>
<name>fs.oss.accessKeySecret</name>
<value>{临时AK_SECRET}</value>
</property>
<property>
<name>fs.oss.buffer.dir</name>
<value>/tmp/oss</value>
</property>
<property>
<name>fs.oss.connection.secure.enabled</name>
<value>false</value>
</property>
<property>
<name>fs.oss.connection.maximum</name>
<value>2048</value>
</property>
</configuration>wget http://apache.communilink.net/spark/spark-3.0.0-preview/spark-3.0.0-preview-bin-hadoop2.7.tgz
tar -xzvf spark-3.0.0-preview-bin-hadoop2.7.tgz在打包之前,需要准备一个docker registry, 可以是docker hub或者是aliyun提供的远程镜像服务.
这里我们使用aliyun的容器镜像服务
docker login --username=lanrish@1416336129779449 registry.us-east-1.aliyuncs.com注:
sudo docker login登录之后,当前用户无法创建镜像.registry.us-east-1.aliyuncs.com这里根据具体选择的地区来决定,默认通过公网访问,我们可以创建k8s集群和镜像服务在同一个地区下(即配置统一的VPC服务),然后在registry后面加一个-vpc,即registry-vpc.us-east-1.aliyuncs.com,这样k8s可以通过内网快速加载容器镜像.cd spark-3.0.0-preview-bin-hadoop2.7HADOOP_CONF_DIR环境变量去自动加载core-site.xml,之所以这么麻烦而不使用ConfigMap,是因为spark 3.0目前存在bug,详见: https://www.jianshu.com/p/d051aa95b241# 构建镜像
./bin/docker-image-tool.sh -r registry.us-east-1.aliyuncs.com/engineplus -t 3.0.0 build
# 发布镜像
docker push registry.us-east-1.aliyuncs.com/engineplus/spark:3.0.0如果需要在镜像中部署额外的依赖环境,则需要使用以下方式:
在spark当前目录spark-3.0.0-preview-bin-hadoop2.7通过Dockerfile的方式构建自定义镜像:
docker build -t registry.us-east-1.aliyuncs.com/spark:3.0.0 -f kubernetes/dockerfiles/spark/Dockerfile可以将自定义的依赖环境定义到kubernetes/dockerfiles/spark/Dockerfile中.
wordcount作业可以从这里clone: https://github.com/i-mine/spark_k8s_wordcount
下载可以直接执行mvn clean package
得到wordcount jar: target/spark_k8s_wordcount-1.0-SNAPSHOT.jar
注: 这种提交方式中,可以上传本地的jar,但是同时需要本地提交环境已经配置过hadoop关于oss的环境.
bin/spark-submit --master k8s://https://192.168.17.175:6443 --deploy-mode cluster --name com.mobvista.dataplatform.WordCount --class com.mobvista.dataplatform.WordCount --conf spark.kubernetes.file.upload.path=oss://mob-emr-test/lei.du/tmp --conf spark.executor.instances=2 --conf spark.kubernetes.container.image=registry.us-east-1.aliyuncs.com/engineplus/spark:3.0.0-oss /home/hadoop/dulei/spark-3.0.0-preview2-bin-hadoop2.7/spark_k8s_wordcount-1.0-SNAPSHOT.jar注: 这种提交方式中,spark依赖的jar只可以是镜像中已经存在的或者是通过远程访问,无法自动将本地的jar上传给spark作业,需要自己手动上传到oss或者s3,且spark镜像中已经存在oss或者s3的访问配置和依赖的jar.
编写spark operator word-count.yaml,这种方式需要提前将jar包打包到镜像中,或者上传到云上.
apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
name: wordcount
namespace: default
spec:
type: Scala
mode: cluster
image: "registry.us-east-1.aliyuncs.com/engineplus/spark:3.0.0-oss"
imagePullPolicy: IfNotPresent
mainClass: com.mobvista.dataplatform.WordCount
mainApplicationFile: "oss://mob-emr-test/lei.du/lib/spark_k8s_wordcount-1.0-SNAPSHOT.jar"
sparkVersion: "3.0.0"
restartPolicy:
type: OnFailure
onFailureRetries: 2
onFailureRetryInterval: 5
onSubmissionFailureRetries: 2
onSubmissionFailureRetryInterval: 10
timeToLiveSeconds: 3600
sparkConf:
"spark.kubernetes.allocation.batch.size": "10"
"spark.eventLog.enabled": "true"
"spark.eventLog.dir": "oss://mob-emr-test/lei.du/tmp/logs"
hadoopConfigMap: oss-hadoop-dir
driver:
cores: 1
memory: "1024m"
labels:
version: 3.0.0
spark-app: spark-wordcount
role: driver
annotations:
k8s.aliyun.com/eci-image-cache: "true"
serviceAccount: spark
executor:
cores: 1
instances: 1
memory: "1024m"
labels:
version: 3.0.0
role: executor
annotations:
k8s.aliyun.com/eci-image-cache: "true"
作业执行过程中我们可以获取ingress-url进行访问WEB UI查看作业执行状态,但是作业执行完毕无法查看:
这里我们使用由Helm chart提供的Spark History Server
GitHub: https://github.com/SnappyDataInc/spark-on-k8s/tree/master/charts/spark-hs?spm=5176.2020520152.0.0.2d5916ddP2xqfh
为了方便,直接通过Aliyun的应用市场进行安装:
应用介绍: https://cs.console.aliyun.com/#/k8s/catalog/detail/incubator_ack-spark-history-server
在创建之前,填写oss相关的配置,然后创建即可:
安装完毕通过查看k8s的server,可以获取到spark history server的访问地址
创建成功后,提交作业的时候,需要添加两条配置:
"spark.eventLog.enabled": "true"
"spark.eventLog.dir": "oss://mob-emr-test/lei.du/tmp/logs"这样提交的作业日志就会存储在OSS.
标签：mat url engine 应用市场 重点 mic 好的 OLE 测试域名
原文地址：https://www.cnblogs.com/lanrish/p/12267623.html
