码迷,mamicode.com
首页 > 编程语言 > 详细

Spring Dataflow批处理框架在OCP上的部署

时间:2018-12-01 20:16:40      阅读:272      评论:0      收藏:0      [点我收藏+]

标签:argument   content   time   project   ace   pen   shel   dash   http   

详细参考

https://donovanmuller.blog/spring-cloud-dataflow-server-openshift/docs/1.2.1.RELEASE/reference/htmlsingle/

 

注意事项:

  • Openshift上需要部署service catalog

 

  • 部署步骤

1.建立项目

#oc login -u admin
#oc new-project scdf --description="Spring Cloud Data Flow"

2.部署模板

官方材料一般都有问题

curl https://raw.githubusercontent.com/donovanmuller/spring-cloud-dataflow-server-openshift/v1.2.1.RELEASE/src/etc/openshift/install-templates.sh | bash

我的做法是,把install-templates.sh下载下来,然后按照sh脚本步骤一步一步部署

#!/usr/bin/env bash

# This script downloads the Data Flow Server for OpenShift templates and uploads them into
# a specified project. The default project is `scdf` as per the Getting Started guide from the reference
# documentation. However, the project can be specified as the first argument to this script.
#
# Usage:
#
# $ ./install-templates.sh [project name]
#
# or alternatively:
#
# $ curl -sL https://github.com/donovanmuller/spring-cloud-dataflow-server-openshift/releases/download/${version}/scdf-openshift-templates.zip \
#   | bash -s [project name] [tag/branch]
#

project=${1:-scdf}
version=${2:-v1.1.0.RELEASE}

echo "Installing OpenShift templates (${version}) into project ‘${project}‘..."

curl -o /tmp/scdf-openshift-templates.zip -sL https://github.com/donovanmuller/spring-cloud-dataflow-server-openshift/releases/download/${version}/scdf-openshift-templates.zip
unzip -o /tmp/scdf-openshift-templates.zip -d /tmp/scdf-openshift-templates

shopt -s nullglob
for template in /tmp/scdf-openshift-templates/*.yaml
do
    echo "Installing template ‘$template‘"
    oc replace --force=true -f $template
done

echo "Adding ‘edit‘ role to ‘scdf‘ Service Account..."

oc policy add-role-to-user edit system:serviceaccount:${project}:scdf

echo "Adding ‘scdf‘ Service Account to the ‘anyuid‘ SCC..."

oc adm policy add-scc-to-user anyuid system:serviceaccount:${project}:scdf

echo "Templates installed."

一看,说白了也就部署一堆templates,因为涉及到好几个镜像,可以按照pullimage.sh文件提供的镜像预先下载

#!/usr/bin/env bash

echo "Pulling images..."

declare -a images=(
  "mysql:5.6"
  "redis:3-alpine"
  "donovanmuller/spring-cloud-dataflow-server-openshift:1.2.0.RELEASE"
  "rabbitmq:3-management"
  "digitalwonderland/zookeeper"
  "wurstmeister/kafka:0.10.2.1"
  )

for((i=0;i<${#images[@]};i++))
do
   echo "Pulling ‘${images[$i]}‘ - `expr $i + 1` of ${#images[@]}"
   docker pull ${images[$i]}
done

因为我的OCP是个离线环境,因此下载完后push到本地的 registry

技术分享图片

 

修改我们要用到的scdf-ephemeral-datasources-kafka-template.yaml,然后oc create -f,可以在catalog中看到

技术分享图片

 

 3.创建实例

保证pod启动

技术分享图片

那个metrics因为没有下载镜像,所以无法启动,暂时不理。

访问 

http://scdf-kafka-scdf.apps.example.com/dashboard/index.html#/apps/apps   出现主界面

技术分享图片

 

 

 授权

oc create -f scdf-sa.yaml
oc policy add-role-to-user edit system:serviceaccount:scdf:scdf
oc adm policy add-scc-to-user anyuid system:serviceaccount:scdf:scdf

 

4.创建任务

  • 启动客户端
[root@master ~]# java -jar spring-cloud-dataflow-shell-1.2.3.RELEASE.jar 
  ____                              ____ _                __
 / ___| _ __  _ __(_)_ __   __ _   / ___| | ___  _   _  __| |
 \___ \| _ \| __| | _ \ / _` | | |   | |/ _ \| | | |/ _` |
  ___) | |_) | |  | | | | | (_| | | |___| | (_) | |_| | (_| |
 |____/| .__/|_|  |_|_| |_|\__, |  \____|_|\___/ \__,_|\__,_|
  ____ |_|    _          __|___/                 __________
 |  _ \  __ _| |_ __ _  |  ___| | _____      __  \ \ \ \ \  | | | |/ _` | __/ _` | | |_  | |/ _ \ \ /\ / /   \ \ \ \ \  | |_| | (_| | || (_| | |  _| | | (_) \ V  V /    / / / / / /
 |____/ \__,_|\__\__,_| |_|   |_|\___/ \_/\_/    /_/_/_/_/_/

1.2.3.RELEASE

Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help".
server-unknown:>
server-unknown:>dataflow config server --uri http://scdf-kafka-scdf.apps.example.com --username user  --password password
Successfully targeted http://scdf-kafka-scdf.apps.example.com

这里注意要用user/password连上,用admin/welcome1是有问题的。

 

  • 注册任务

按照官方文档,又是有问题的,后来自己下载下来然后用文件方式导入

dataflow:>app import --uri http://bit.ly/1-0-1-GA-task-applications-maven

因为只用到一个任务,所以先修改

[root@master ~]# cat timestamp1.task 
task.timestamp=docker:docker-registry.default.svc:5000/scdf/timestamp-task:latest

导入

dataflow:>app import --uri file:////root/timestamp1.task
Successfully registered applications: [task.timestamp]
  • 创建任务并运行
dataflow:>task create task1 --definition "timestamp"
Created new task task1
dataflow:>task launch task1
Launched task task1

在界面上看到一个task

技术分享图片

 

在ocp console上也看到这些task1的pod

技术分享图片

 

Spring Dataflow批处理框架在OCP上的部署

标签:argument   content   time   project   ace   pen   shel   dash   http   

原文地址:https://www.cnblogs.com/ericnie/p/10050464.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!