ONOS HA 部署流程
高可靠性的网络环境在生产中随处可见，ONOS的一大亮点是支持集群。为了能够快速的搭建起ONOS的集群环境，官网提供了基于docker的部署镜像。当我们部署完成docker环境以后，可以很方便的使用docker来部署一个ONOS的集群。[关于docker的环境部署，参考：http://www.cnblogs.com/BurlyLuo/p/8283815.html]
ONOS使用Hazelcast架构实现对集群成员的管理，在集群模式下，网络中的控制器共享状态信息，共同管理网络中的设备（交换机）。使用ONOS建立集群的时候会将集群中的设备都默认加入到224.2.2.3:54327组播组中，组播地址可以通过修改hazelcast.xml文件指定。
1.基础平台搭建。
本实验是在上一篇博客[SDN模拟器ONOS Install Guide]中提到的ONOS环境中利用容器技术做出的另一套ONOS的集群环境，CPU和memory配置均未更改，发现运行没有任何问题。我在实际中加了一块桥接网卡，配置的地址：
192.168.43.119//方便通过CRT软件登录实现copy/paste//可不做，直接在图形化桌面操作即可。
//但是必须的使用distributed/distributed 用户名/密码登录CLI，否则会导致部分命令无法使用。//
Like This:
----------------------------------------------------------------------
Last login: Mon Feb 19 18:20:56 2018 from luowei
tutorial1@mininet-vm:~$ ssh distributed@192.168.43.119 //用户名：distributed
distributed@192.168.43.119‘s password: //密码为：distributed
Welcome to Ubuntu 14.10 (GNU/Linux 3.16.0-23-generic x86_64)
* Documentation: https://help.ubuntu.com/
Your Ubuntu release is not supported anymore.
For upgrade information, please visit:
http://www.ubuntu.com/releaseendoflife
2.使用docker部署ONOS集群
2.1查看该系统中的docker images，如果没有可以使用docker pull sdnsub/onos 来拉取对应的ONOS image。
distributed@mininet-vm:~$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
onos/tutorial-dist latest cc5df7fe591e 3.141949 years ago 661.5 MB
ubuntu-upstart 14.10 ac87bc62880a 3.184969 years ago 259.4 MB
distributed@mininet-vm:~$
2.2在该系统上运行三个docker container。
distributed@mininet-vm:~$ sudo docker run -t -P -i -d --name onos-1 onos/tutorial-dist
distributed@mininet-vm:~$ sudo docker run -t -P -i -d --name onos-2 onos/tutorial-dist
distributed@mininet-vm:~$ sudo docker run -t -P -i -d --name onos-3 onos/tutorial-dist
查看运行的container:
distributed@mininet-vm:~$ sudo docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b5f408276c70 onos/tutorial-dist:latest "/sbin/init" 22 minutes ago Up 22 minutes 0.0.0.0:49159->22/tcp, 0.0.0.0:49160->6633/tcp, 0.0.0.0:49161->8181/tcp onos-3
34e068d7032b onos/tutorial-dist:latest "/sbin/init" 22 minutes ago Up 22 minutes 0.0.0.0:49156->22/tcp, 0.0.0.0:49157->6633/tcp, 0.0.0.0:49158->8181/tcp onos-2
1f3a3cdc3b63 onos/tutorial-dist:latest "/sbin/init" 23 minutes ago Up 23 minutes 0.0.0.0:49153->22/tcp, 0.0.0.0:49154->6633/tcp, 0.0.0.0:49155->8181/tcp onos-1
2.3检查docker的环境是否正确。
distributed@mininet-vm:~$ cell docker
ONOS_CELL=docker
OCI=172.17.0.2
OC1=172.17.0.2
OC2=172.17.0.3
OC3=172.17.0.4
OCN=localhost
ONOS_FEATURES=webconsole,onos-api,onos-core,onos-cli,onos-rest,onos-gui,onos-openflow,onos-app-fwd,onos-app-proxyarp,onos-app-mobility
ONOS_USER=root
ONOS_NIC=172.17.0.*
-------------------------------------------------------------------------------------------------------------------------------------------
该步骤非常关键。跳过可能会导致$OC的值没有及时更新而发生错误。我们知道在docker默认的情况下，docker0为172.17.0.0/16的网段，所以默认情况下刚才创建的三个containers
所获得地址为172.17.0.x的地址。我们可以进入container中查看地址。
-------------------------------------------------------------------------------------------------------------------------------------------
2.4设置无密码访问
系统镜像提供了该脚本，我们可以直接使用。[其中默认的密码为：onosrocks]
distributed@mininet-vm:~$ onos-push-keys $OC1
The authenticity of host ‘172.17.0.2 (172.17.0.2)‘ can‘t be established.
ECDSA key fingerprint is a7:83:a2:2e:43:e3:39:19:1f:a5:05:ae:80:74:26:b6.
Are you sure you want to continue connecting (yes/no)? yes
root@172.17.0.2‘s password:
root@172.17.0.2‘s password:
distributed@mininet-vm:~$ onos-push-keys $OC2
The authenticity of host ‘172.17.0.3 (172.17.0.3)‘ can‘t be established.
ECDSA key fingerprint is a7:83:a2:2e:43:e3:39:19:1f:a5:05:ae:80:74:26:b6.
Are you sure you want to continue connecting (yes/no)? yes
root@172.17.0.3‘s password:
root@172.17.0.3‘s password:
distributed@mininet-vm:~$ onos-push-keys $OC3
The authenticity of host ‘172.17.0.4 (172.17.0.4)‘ can‘t be established.
ECDSA key fingerprint is a7:83:a2:2e:43:e3:39:19:1f:a5:05:ae:80:74:26:b6.
Are you sure you want to continue connecting (yes/no)? yes
root@172.17.0.4‘s password:
root@172.17.0.4‘s password:
2.5下载对应的onos软件包
distributed@mininet-vm:~$ onos-package
-rw-rw-r-- 1 distributed distributed 41940635 Feb 19 18:07 /tmp/onos-1.0.0.distributed.tar.gz
2.6在刚才部署的三个docker container中安装onos。
distributed@mininet-vm:~$ onos-install $OC1
onos start/running, process 315
distributed@mininet-vm:~$ onos-install $OC2
onos start/running, process 317
distributed@mininet-vm:~$ onos-install $OC3
onos start/running, process 315
2.7进入onos的cli
For OC1
--------------------------------------------------------------
ONOS-1
--------------------------------------------------------------
distributed@mininet-vm:~$ onos -w $OC1
Connection to 172.17.0.2 closed.
client: JAVA_HOME not set; results may vary
Logging in as karaf
Welcome to Open Network Operating System (ONOS)!
____ _ ______ ____
/ __ \/ |/ / __ \/ __/
/ /_/ / / /_/ /\ \
\____/_/|_/\____/___/
Hit ‘<tab>‘ for a list of available commands
and ‘[cmd] --help‘ for help on a specific command.
Hit ‘<ctrl-d>‘ or type ‘system:shutdown‘ or ‘logout‘ to shutdown ONOS.
For OC2
--------------------------------------------------------------
ONOS-2
--------------------------------------------------------------
Last login: Mon Feb 19 17:14:09 2018 from luowei
tutorial1@mininet-vm:~$
tutorial1@mininet-vm:~$ ssh distributed192.168.43.19
^C
tutorial1@mininet-vm:~$ ssh distributed192.168.43.119
^C
tutorial1@mininet-vm:~$ ssh distributed@192.168.43.119
distributed@192.168.43.119‘s password:
Welcome to Ubuntu 14.10 (GNU/Linux 3.16.0-23-generic x86_64)
* Documentation: https://help.ubuntu.com/
Your Ubuntu release is not supported anymore.
For upgrade information, please visit:
http://www.ubuntu.com/releaseendoflife
New release ‘15.04‘ available.
Run ‘do-release-upgrade‘ to upgrade to it.
Last login: Mon Feb 19 18:17:00 2018 from 192.168.43.119
distributed@mininet-vm:~$ onos -w $OC2
Connection to 172.17.0.3 closed.
client: JAVA_HOME not set; results may vary
Logging in as karaf
Welcome to Open Network Operating System (ONOS)!
____ _ ______ ____
/ __ \/ |/ / __ \/ __/
/ /_/ / / /_/ /\ \
\____/_/|_/\____/___/
Hit ‘<tab>‘ for a list of available commands
and ‘[cmd] --help‘ for help on a specific command.
Hit ‘<ctrl-d>‘ or type ‘system:shutdown‘ or ‘logout‘ to shutdown ONOS.
onos>
For OC3
------------------------------------------------------------
ONOS-3
------------------------------------------------------------
Last login: Mon Feb 19 18:11:07 2018 from luowei
tutorial1@mininet-vm:~$ ssh distributed@192.168.43.19
ssh: connect to host 192.168.43.19 port 22: No route to host
tutorial1@mininet-vm:~$ ssh distributed@192.168.43.119
The authenticity of host ‘192.168.43.119 (192.168.43.119)‘ can‘t be established.
ECDSA key fingerprint is b2:89:b0:31:07:05:f3:28:88:be:19:71:79:9a:f5:21.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘192.168.43.119‘ (ECDSA) to the list of known hosts.
distributed@192.168.43.119‘s password:
Welcome to Ubuntu 14.10 (GNU/Linux 3.16.0-23-generic x86_64)
* Documentation: https://help.ubuntu.com/
Your Ubuntu release is not supported anymore.
For upgrade information, please visit:
http://www.ubuntu.com/releaseendoflife
New release ‘15.04‘ available.
Run ‘do-release-upgrade‘ to upgrade to it.
Last login: Mon Feb 19 18:04:16 2018 from 192.168.43.119
distributed@mininet-vm:~$ onos -w $OC3
Connection to 172.17.0.4 closed.
client: JAVA_HOME not set; results may vary
Logging in as karaf
Welcome to Open Network Operating System (ONOS)!
____ _ ______ ____
/ __ \/ |/ / __ \/ __/
/ /_/ / / /_/ /\ \
\____/_/|_/\____/___/
Hit ‘<tab>‘ for a list of available commands
and ‘[cmd] --help‘ for help on a specific command.
Hit ‘<ctrl-d>‘ or type ‘system:shutdown‘ or ‘logout‘ to shutdown ONOS.
onos>
至此SND的控制器ONOS已经部署完成，有了前边的经验，这里有可能需要自己安装一些feature，比如ONOS GUI的feature等等，需要特别的注意。
3.启动MIninet。
tutorial1@mininet-vm:~$ ssh distributed@192.168.43.119
distributed@192.168.43.119‘s password:
Welcome to Ubuntu 14.10 (GNU/Linux 3.16.0-23-generic x86_64)
* Documentation: https://help.ubuntu.com/
Your Ubuntu release is not supported anymore.
For upgrade information, please visit:
http://www.ubuntu.com/releaseendoflife
New release ‘15.04‘ available.
Run ‘do-release-upgrade‘ to upgrade to it.
Last login: Mon Feb 19 18:17:10 2018 from 192.168.43.119
distributed@mininet-vm:~$ cd onos-byon && ./startmn.sh
*** Creating network
*** Adding hosts:
h11 h12 h13 h14 h15 h16 h21 h22 h23 h24 h25 h26 h31 h32 h33 h34 h35 h36 h41 h42 h43 h44 h45 h46
*** Adding switches:
s1 s2 s11 s12 s13 s14
*** Adding links:
(h11, s11) (h12, s11) (h13, s11) (h14, s11) (h15, s11) (h16, s11) (h21, s12) (h22, s12) (h23, s12) (h24, s12) (h25, s12) (h26, s12) (h31, s13) (h32, s13) (h33, s13) (h34, s13) (h35, s13) (h36, s13) (h41, s14) (h42, s14) (h43, s14) (h44, s14) (h45, s14) (h46, s14) (s1, s2) (s11, s1) (s11, s2) (s12, s1) (s12, s2) (s13, s1) (s13, s2) (s14, s1) (s14, s2)
*** Configuring hosts
h11 h12 h13 h14 h15 h16 h21 h22 h23 h24 h25 h26 h31 h32 h33 h34 h35 h36 h41 h42 h43 h44 h45 h46
*** Starting controller
c0 c0 c0
*** Starting 6 switches
s1 s2 s11 s12 s13 s14
*** Starting CLI:
mininet>
mininet> pingall
*** Ping: testing ping reachability
h11 -> h12 h13 h14 h15 h16 h21 h22 h23 h24 h25 h26 h31 h32 h33 h34 h35 h36 h41 h42 h43 h44 h45 h46
h12 -> h11 h13 h14 h15 h16 h21 h22 h23 h24 h25 h26 h31 h32 h33 h34 h35 h36 h41 h42 h43 h44 h45 h46
h13 -> h11 h12 h14 h15 h16 h21 h22 h23 h24 h25 h26 h31 h32 h33 h34 h35 h36 h41 h42 h43 h44 h45 h46
h14 -> h11 h12 h13 h15 h16 h21 h22 h23 h24 h25 h26 h31 h32 h33 h34 h35 h36 h41 h42 h43 h44 h45 h46
h15 -> h11 h12 h13 h14 h16 h21 h22 h23 h24 h25 h26 h31 h32 h33 h34 h35 h36 h41 h42 h43 h44 h45 h46
h16 -> h11 h12 h13 h14 h15 h21 h22 h23 h24 h25 h26 h31 h32 h33 h34 h35 h36 h41 h42 h43 h44 h45 h46
h21 -> h11 h12 h13 h14 h15 h16 h22 h23 h24 h25 h26 h31 h32 h33 h34 h35 h36 h41 h42 h43 h44 h45 h46
h22 -> h11 h12 h13 h14 h15 h16 h21 h23 h24 h25 h26 h31 h32 h33 h34 h35 h36 h41 h42 h43 h44 h45 h46
h23 -> h11 h12 h13 h14 h15 h16 h21 h22 h24 h25 h26 h31 h32 h33 h34 h35 h36 h41 h42 h43 h44 h45 h46
h24 -> h11 h12 h13 h14 h15 h16 h21 h22 h23 h25 h26 h31 h32 h33 h34 h35 h36 h41 h42 h43 h44 h45 h46
h25 -> h11 h12 h13 h14 h15 h16 h21 h22 h23 h24 h26 h31 h32 h33 h34 h35 h36 h41 h42 h43 h44 h45 h46
h26 -> h11 h12 h13 h14 h15 h16 h21 h22 h23 h24 h25 h31 h32 h33 h34 h35 h36 h41 h42 h43 h44 h45 h46
h31 -> h11 h12 h13 h14 h15 h16 h21 h22 h23 h24 h25 h26 h32 h33 h34 h35 h36 h41 h42 h43 h44 h45 h46
h32 -> h11 h12 h13 h14 h15 h16 h21 h22 h23 h24 h25 h26 h31 h33 h34 h35 h36 h41 h42 h43 h44 h45 h46
h33 -> h11 h12 h13 h14 h15 h16 h21 h22 h23 h24 h25 h26 h31 h32 h34 h35 h36 h41 h42 h43 h44 h45 h46
h34 -> h11 h12 h13 h14 h15 h16 h21 h22 h23 h24 h25 h26 h31 h32 h33 h35 h36 h41 h42 h43 h44 h45 h46
h35 -> h11 h12 h13 h14 h15 h16 h21 h22 h23 h24 h25 h26 h31 h32 h33 h34 h36 h41 h42 h43 h44 h45 h46
h36 -> h11 h12 h13 h14 h15 h16 h21 h22 h23 h24 h25 h26 h31 h32 h33 h34 h35 h41 h42 h43 h44 h45 h46
h41 -> h11 h12 h13 h14 h15 h16 h21 h22 h23 h24 h25 h26 h31 h32 h33 h34 h35 h36 h42 h43 h44 h45 h46
h42 -> h11 h12 h13 h14 h15 h16 h21 h22 h23 h24 h25 h26 h31 h32 h33 h34 h35 h36 h41 h43 h44 h45 h46
h43 -> h11 h12 h13 h14 h15 h16 h21 h22 h23 h24 h25 h26 h31 h32 h33 h34 h35 h36 h41 h42 h44 h45 h46
h44 -> h11 h12 h13 h14 h15 h16 h21 h22 h23 h24 h25 h26 h31 h32 h33 h34 h35 h36 h41 h42 h43 h45 h46
h45 -> h11 h12 h13 h14 h15 h16 h21 h22 h23 h24 h25 h26 h31 h32 h33 h34 h35 h36 h41 h42 h43 h44 h46
h46 -> h11 h12 h13 h14 h15 h16 h21 h22 h23 h24 h25 h26 h31 h32 h33 h34 h35 h36 h41 h42 h43 h44 h45
*** Results: 0% dropped (552/552 received)
mininet> h12 ping h13
PING 10.0.0.3 (10.0.0.3) 56(84) bytes of data.
64 bytes from 10.0.0.3: icmp_seq=1 ttl=64 time=20.8 ms
64 bytes from 10.0.0.3: icmp_seq=2 ttl=64 time=0.573 ms
64 bytes from 10.0.0.3: icmp_seq=3 ttl=64 time=0.064 ms
^C
--- 10.0.0.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2005ms
rtt min/avg/max/mdev = 0.064/7.157/20.835/9.674 ms
mininet>
mininet> nodes
available nodes are:
c0 c0 c0 h11 h12 h13 h14 h15 h16 h21 h22 h23 h24 h25 h26 h31 h32 h33 h34 h35 h36 h41 h42 h43 h44 h45 h46 s1 s11 s12 s13 s14 s2
mininet>
4.安装完ONOS的feature之后，我们可以在浏览器中打开部署好的环境。
在ONOS-1上查看
onos> summary
node=172.17.0.2, version=1.0.0.distributed~2018/02/19@18:06
nodes=3, devices=0, links=0, hosts=0, SCC(s)=0, paths=0, flows=0, intents=0
onos> masters
172.17.0.2: 0 devices
172.17.0.3: 0 devices
172.17.0.4: 0 devices
onos> devices
onos> roles
onos>
激活Mininit以后：
onos> devices
id=of:0000000000000001, available=true, role=MASTER, type=SWITCH, mfr=Nicira, Inc., hw=Open vSwitch, sw=2.1.3, serial=None, protocol=OF_10
id=of:0000000000000002, available=true, role=MASTER, type=SWITCH, mfr=Nicira, Inc., hw=Open vSwitch, sw=2.1.3, serial=None, protocol=OF_10
id=of:000000000000000b, available=true, role=MASTER, type=SWITCH, mfr=Nicira, Inc., hw=Open vSwitch, sw=2.1.3, serial=None, protocol=OF_10
id=of:000000000000000c, available=true, role=STANDBY, type=SWITCH, mfr=Nicira, Inc., hw=Open vSwitch, sw=2.1.3, serial=None, protocol=OF_10
id=of:000000000000000d, available=true, role=STANDBY, type=SWITCH, mfr=Nicira, Inc., hw=Open vSwitch, sw=2.1.3, serial=None, protocol=OF_10
id=of:000000000000000e, available=true, role=STANDBY, type=SWITCH, mfr=Nicira, Inc., hw=Open vSwitch, sw=2.1.3, serial=None, protocol=OF_10
onos>
onos> balance-masters //做负载均衡//
onos> masters
172.17.0.2: 2 devices
of:0000000000000002
of:000000000000000b
172.17.0.3: 2 devices
of:000000000000000c
of:000000000000000e
172.17.0.4: 2 devices
of:0000000000000001
of:000000000000000d
onos>
onos> nodes
id=172.17.0.2, address=172.17.0.2:9876, state=ACTIVE *
id=172.17.0.3, address=172.17.0.3:9876, state=ACTIVE
id=172.17.0.4, address=172.17.0.4:9876, state=ACTIVE
当然这里主要是针对于集群环境下的部署的一个说明，如果我们停掉一个ONOS节点 ，我们会在ONOS的GUi上看到相应的控制器退出集群，剩下的继续提供服务。可自己实验，这里就不做赘述。
这里顺便提一下ONOS和ODL的比较。节选自知乎用户:常景路.
ONOS和ODL分别由运营商和厂商主导，所代表的利益不同，也就分别选择了两种不同的SDN演进方式。前者更贴近于SDN诞生之初时狭义的SDN概念，即通过OpenFlow将控制平面和转发平面完全分离，网络设备只是进行转发的黑盒子，通过Controller完成一切计算。ONOS所选择的理念与运营商自己的利益息息相关，只有将控制能力拿到自己手里，才能在整条产业链上逐步摆脱设备厂商的控制。通过使用更为廉价的转发设备替代原有的厂商设备，一方面在眼下增加自己与设备厂商的议价砝码，另一方面长远看能大大降低网络的建设和维护成本。相比较而言，ODL则采取了更为平缓的SDN演进方式，从理念上更为贴近广义的SDN，即不局限于OpenFlow协议，不局限于完全将控制平面从转发设备上剥离，通过已有的网络协议将部分的控制逻辑放到Controller上。这样的理念使广义的SDN技术的落地更容易成为现实，一方面通过保护运营商、企业等设备厂商客户的既有投资，使客户可以真正感受到SDN技术的实际效果。另一方面，通过在现有设备上扩展已有的网络协议，厂商能够使自己的设备在不用伤筋动骨就能保有竞争力，避免自己在SDN的革命中被迅速甩下。从技术上讲，SDN Controller实际上解决的是南向与设备的通信问题和北向向APP提供的资源问题，网络运营者根据自己网络的业务特点提出的控制逻辑则需要开发APP来实现。从南向接口上看，ONOS目前成熟的南向接口只有OpenFlow，而ODL Helium版则支持OpenFlow、OVS-DB、MP-BGP、PCEP、NETCONF/YANG等极为丰富的南向接口以连接不同类型的设备。从北向接口上看，ODL采用的MD-SAL使得设备资源可以通过YANG model直接转换为RESTConf API，而ONOS还在某种程度上停留在ODL最初版本使用的AD-SAL架构，API需要在plugin设计时单独考量。当然除此之外，Controller的性能与Scale out也是必须面对的问题。对此，ONOS确实抓住了ODL尚未解决的问题，从一开始就从这两方面抢占先机，拨人眼球。不过从二者实现上都采用了JAVA的Karaf框架来看，性能与Scale out问题在根本上也不会存在先天的差别，面对海量计算采用Cluster会是最终的解决方法，而实际上两个控制器都提供了相应的Cluster部署方案。唯一的问题可能是ODL还需要应对多种南向接口带来的额外消耗，但ODL提供的是南向接口的可选能力，实际部署上也很少会出现多种协议共存的情况。
5.如有疑问。
########################################################################
#Date:2018-03-08 Author:BurlyLuo#
#Mail:olaf.luo@foxmail.com Version:v1.0 #
########################################################################
非商业用途转载无需作者授权，但务必在文章标题下面注明作者 罗伟（Burly Luo）以及可点击的本博客地址超级链接 http://www.cnblogs.com/BurlyLuo/，谢谢合作.
