标签：epo   ast   gitlab   path   dwr   col   copy   14.   shadow   

本文介绍在k8s环境中进行jenkins server的部署和配置。Jenkins是一个开源的、功能强大的持续集成和持续构建工具，采用master和salve架构，我们通过将jenkins集成环境部署在k8s集群中，可以实现jenkins slave按需创建、动态的伸缩。同时也提供了在k8s环境中应用的持续部署解决方案。

一、准备docker镜像文件

1、编译jenkins server docker镜像，默认的jenkis镜像已包含jdk，版本为1.8.0_171

# cat dockerfile
FROM jenkins
MAINTAINER ylw@fjhb.cn
ENV MAVEN_HOME /usr/local/maven
ENV JAVA_HOME  /usr/local/java
ENV CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
ENV PATH ${JAVA_HOME}/bin:${MAVEN_HOME}/bin:${PATH}

COPY apache-maven-3.5.4 /usr/local/maven
USER root
RUN mkdir -p /usr/local/maven/repository  &&  ln -s /usr/java/jdk1.8.0_171 /usr/local/java

# docker build -t harbor.59iedu.com/fjhb/jenkins:2018-08-12-v1 . 
# docker push harbor.59iedu.com/fjhb/jenkins:2018-08-12-v1

2、编译jenkins slave镜像
可以根据实际情况配置maven内网私服nexus，私服可以避免编译过程中通过公网下载依赖的jar包，配置私服需要把对应的setting.xml文件打包到apache-maven-3.5.4/conf目录下；
libltdl.so.7文件的获取路径为操作系统路径/usr/lib64/libltdl.so.7（实际上是个软链接，需要copy出来重命名）
slave.jar文件的获取路径为http://jenkins-server/jnlpJars/slave.jar

# cat Dockerfile
FROM openshift/base-centos7
MAINTAINER ylw@fjhb.cn
COPY apache-maven-3.5.4 /usr/local/maven
COPY jdk1.8.0_171       /usr/local/java
COPY kubectl            /usr/local/bin/kubectl
COPY libltdl.so.7 /usr/lib64/libltdl.so.7
COPY slave.jar /usr/share/jenkins/slave.jar 
COPY jenkins-slave /usr/local/bin/jenkins-slave

ENV HOME /home/jenkins
ENV AGENT_WORKDIR=/home/jenkins/agent
ENV JAVA_HOME /usr/local/java
ENV MAVEN_HOME /usr/local/maven/
ENV CLASSPATH .:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
ENV PATH ${JAVA_HOME}/bin:${MAVEN_HOME}/bin:${PATH} 
ENV MAVEN_CONFIG "$USER_HOME_DIR/.m2"

RUN  chmod 755 /usr/share/jenkins   && chmod 644 /usr/share/jenkins/slave.jar 

RUN mkdir -p /home/jenkins/.jenkins  && mkdir -p ${AGENT_WORKDIR}  && yum -y install git subversion sshpass

VOLUME /home/jenkins/.jenkins
VOLUME ${AGENT_WORKDIR}
WORKDIR /home/jenkins

USER root
ENTRYPOINT ["jenkins-slave"]

# docker build -t harbor.59iedu.com/fjhb/jenkins-slave-toolkit:2018-08-10-v1 . 
# docker push harbor.59iedu.com/fjhb/jenkins-slave-toolkit:2018-08-10-v1

二、创建jenkins server

1、创建pv和pvc

# cat pv.yaml 
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-master-vol
spec:
  capacity:
    storage: 5Gi 
  accessModes:
  - ReadWriteMany 
  nfs: 
    path: /home/jenkins
    server: 192.168.115.6
  persistentVolumeReclaimPolicy: Recycle 
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: maven-repository
spec:
  capacity:
    storage: 5Gi
  accessModes:
  - ReadWriteMany
  nfs:
    path: /home/maven
    server: 192.168.115.6
  persistentVolumeReclaimPolicy: Recycle

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: jenkins-master-claim
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: maven-repository-claim
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi

2、创建deployment和service

# cat deploy.yaml 
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins-master
spec:
  template:
    metadata:
      labels:
        name: jenkins-master
    spec:
      securityContext:
        fsGroup: 1000
      containers:
        - name: jenkins-master
          image: harbor.59iedu.com/fjhb/jenkins:2018-08-12-v1
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
              name: http
            - containerPort: 50000
              name: agent
          volumeMounts:
            - name: jenkins-master-vol
              mountPath: /var/jenkins_home
            - name: maven-repository
              mountPath: /opt/maven/repository
            - name: docker
              mountPath: /usr/bin/docker
            - name: docker-sock
              mountPath: /var/run/docker.sock
      volumes:
        - name: jenkins-master-vol
          persistentVolumeClaim:
            claimName: jenkins-master-claim
        - name: maven-repository
          persistentVolumeClaim:
            claimName: maven-repository-claim
        - name: docker
          hostPath:
            path: /usr/bin/docker
        - name: docker-sock
          hostPath:
            path: /var/run/docker.sock
      serviceAccount: "jenkins-master"
      imagePullSecrets:
        - name: harborsecret

---
apiVersion: v1
kind: Service
metadata:
  name: jenkins-master
spec:
  type: NodePort
  ports:
    - port: 8080
      name: http
      targetPort: 8080
      nodePort: 8452
    - port: 50000
      name: agent
      nodePort: 50000
      targetPort: 50000 
  selector:
    name: jenkins-master

3、rbac授权

# cat sa.yaml 
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-master
  namespace: default

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins-master

rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: jenkins-master

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins-master
subjects:
- kind: ServiceAccount
  name: jenkins-master
  namespace: default

4、default sa的rbac授权

# cat default-sa.yaml 
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: default-role

rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: default-rolebinding

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: default-role 
subjects:
- kind: ServiceAccount
  name: default
  namespace: default

三、初始化jenkins server

1、通过秘钥解锁jenkins（本文为了方便局域网其他主机访问，在vmware上配置了nat规则）

2、配置代理


3、安装插件


4、创建管理员账号

四、配置jenkins server

1、系统管理 —— 系统设置 —— 新增一个云”kubernetes”

Kubernetes URL: 输入api-server的地址
Jenkins URL: 输入jenkins server的服务名,端口8080
Jenkins Tunnel: 指的是slave连接master的端口，默认是50000

上图pod的模板名称为jenkins-slave，Container的模板名称为jnlp。这里有非常重要的两点要注意：
当Container的模板名称为jnlp的时候，jenkins-slave才会使用下面配置的docker镜像来启动pod，如果不为jnlp，则会使用默认的镜像jenkins/jnlp-slave:alpine

当使用自定义的docker镜像来启动jenkins slave pod的时候，下面的command to run（默认值是 sh -c）和arguments to pass to the command(默认值是cat)两个值需要清空。否则会出现jenkins slave jnlp连接不上master的情况，尝试100次连接之后销毁pod，然后再创建一个pod继续尝试连接，无限循环。

2、系统管理 —— Configure Global Security
确认jnlp agent的端口默认为50000，如果有修改，要保障这里的配置及前面部署deployment、service的端口配置、前文的云环境Jenkins Tunnel设置保持一致

3、系统管理 —— Global Tool Configuration
在这里设置对应的工具及环境变量，为了避免不必要的问题，前面通过dockerfile把jenkins server 和jenkins slave的环境变量调整成一致, java目录通过软连接的方式实现。

4、系统管理 —— 管理插件
推荐安装的几个插件：maven、 gitlab 、subversion、pipeline、Kubernetes Continuous Deploy、Publish Over SSH

完成插件安装后需要对jenkins server进行重启操作，可以点击“系统管理 ”——“准备关机”来完成重启操作，至此我们就完成了jenkins server在k8s环境中的部署和配置工作，下文开始介绍使用jenkins完成项目构建和发布。

K8S集群中部署jenkins

标签：epo   ast   gitlab   path   dwr   col   copy   14.   shadow   

原文地址：http://blog.51cto.com/ylw6006/2159769