码迷,mamicode.com
首页 > 其他好文 > 详细

DVR分布式路由

时间:2015-10-26 00:20:02      阅读:391      评论:0      收藏:0      [点我收藏+]

标签:

1. 背景

  没有使用DVR的场景:

  技术分享

  从图中可以明显看到东西向和南北向的流量会集中到网络节点,这会使网络节点成为瓶颈。

  如果启用DVR,如下图:

  技术分享

 

  对于东西向的流量, 流量会直接在计算节点之间传递。

  对于南北向的流量,如果有floating ip,流量就直接走计算节点。如果没有floating ip,则会走网络节点。
 

 2.部署以及流量走向

  技术分享

   技术分享

  2.1东西向流量

  VM1 (10.0.1.5 Net1) ping VM2 (10.0.2.5 Net2)

   1) VM1 (10.0.1.5) -> qr (10.0.1.1)

    VM1 根据默认路由发送arp(广播)请求qr网关的地址,请求到网关地址后,icmp报文走向qr口。

    (关于报文格式的一点解释,当VM1 ping VM2时,报文的源/目的IP始终不变,报文的源/目的MAC则会根据不同的路段而变化。

    同时,br-tun网桥会丢弃目的地址是interface_distributed接口的arp广播,不至于让不必要的流量流向外面:

# ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):  
...
cookie=0x0, duration=64720.432s, table=1, n_packets=4, n_bytes=168, idle_age=64607, priority=3,arp,dl_vlan=1,arp_tpa=10.0.1.1 actions=drop
...

   2)qr  (10.0.1.1) -> qr (10.0.2.1)

    进入qrouter namespace后,利用linux内核的高级路由功能,查看路由规则。

# ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa ip rule  
0: from all lookup local  
32766: from all lookup main  
32767: from all lookup default  
32768: from 10.0.1.5 lookup 16  
32769: from 10.0.2.3 lookup 16  
167772417: from 10.0.1.1/24 lookup 167772417  
167772417: from 10.0.1.1/24 lookup 167772417  
167772673: from 10.0.2.1/24 lookup 167772673 

    先查看main表:

# ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa ip route list table main  
10.0.1.0/24 dev qr-ddbdc784-d7 proto kernel scope link src 10.0.1.1  
10.0.2.0/24 dev qr-001d0ed9-01 proto kernel scope link src 10.0.2.1  
169.254.31.28/31 dev rfp-0fbb351e-a proto kernel scope link src 169.254.31.28

    在main表中满足以上路由,因此会从另一个qr口出去。(Q1:不同计算节点的同一子网下qr口ip是相同的吗?)

   3)qr -> br-int   

  之后需要去查询10.0.2.5的MAC地址, MAC是由neutron使用静态ARP的方式设定的,由于Neutron知道所有VM的信息,因此他可以事先设定好静态ARP:

# ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa ip nei  
10.0.1.5 dev qr-ddbdc784-d7 lladdr fa:16:3e:da:75:6d PERMANENT  
10.0.2.3 dev qr-001d0ed9-01 lladdr fa:16:3e:a4:fc:98 PERMANENT  
10.0.1.6 dev qr-ddbdc784-d7 lladdr fa:16:3e:9f:55:67 PERMANENT  
10.0.2.2 dev qr-001d0ed9-01 lladdr fa:16:3e:13:55:66 PERMANENT  
10.0.2.5 dev qr-001d0ed9-01 lladdr fa:16:3e:51:99:b8 PERMANENT 
10.0.1.4 dev qr-ddbdc784-d7 lladdr fa:16:3e:da:e3:6e PERMANENT  
10.0.1.7 dev qr-ddbdc784-d7 lladdr fa:16:3e:14:b8:ec PERMANENT  
169.254.31.29 dev rfp-0fbb351e-a lladdr 42:0d:9f:49:63:c6 STALE

  此时,报文进入br-int,根据table 0 进行normal转发:

cookie=0x0, duration=16440.644s, table=0, n_packets=1074, n_bytes=104318, idle_age=8917, priority=1 actions=NORMAL

  normal动作则表示根据OVS fdb表项匹配目的MAC地址,从而决定该报文要往哪个端口发送。如果没有该MAC的fdb表项记录,则进行泛洪,对除了报文进来的端口以外的所有同属于一个vlan的端口发送该报文。例如:

# ovs-appctl fdb/show br-int
 port  VLAN  MAC                Age
LOCAL     0  da:91:42:cd:fb:44   18
   18     0  52:54:00:a9:b8:b0    0
   19     0  52:54:00:a9:b8:b1    0

  因此如果此时VM2也在该compute node上,则VM2也会直接收到该报文,不需要走br-tun(有了VM2的MAC fdb表项记录后)。否则,继续往br-tun走。

  4)br-int -> br-tun -> 出compute node 1

  然后报文从br-int进入br-tun匹配流表:

 cookie=0x0, duration=66172.51s, table=0, n_packets=58, n_bytes=5731, idle_age=20810, hard_age=65534, priority=1,in_port=3 actions=resubmit(,4)
 cookie=0x0, duration=67599.526s, table=0, n_packets=273, n_bytes=24999, idle_age=1741, hard_age=65534, priority=1,in_port=1 actions=resubmit(,1)
 cookie=0x0, duration=64437.052s, table=0, n_packets=28, n_bytes=2980, idle_age=20799, priority=1,in_port=4 actions=resubmit(,4)
 cookie=0x0, duration=67601.704s, table=0, n_packets=5, n_bytes=390, idle_age=65534, hard_age=65534, priority=0 actions=drop
 cookie=0x0, duration=66135.811s, table=1, n_packets=140, n_bytes=13720, idle_age=65534, hard_age=65534, priority=1,dl_vlan=1,dl_src=fa:16:3e:66:13:af actions=mod_dl_src:fa:16:3f:fe:49:e9,resubmit(,2)
 cookie=0x0, duration=64082.141s, table=1, n_packets=2, n_bytes=200, idle_age=64081, priority=1,dl_vlan=2,dl_src=fa:16:3e:69:b4:05 actions=mod_dl_src:fa:16:3f:fe:49:e9,resubmit(,2)
 cookie=0x0, duration=66135.962s, table=1, n_packets=1, n_bytes=98, idle_age=65301, hard_age=65534, priority=2,dl_vlan=1,dl_dst=fa:16:3e:66:13:af actions=drop 
 cookie=0x0, duration=64082.297s, table=1, n_packets=0, n_bytes=0, idle_age=64082, priority=2,dl_vlan=2,dl_dst=fa:16:3e:69:b4:05 actions=drop
 cookie=0x0, duration=66136.115s, table=1, n_packets=4, n_bytes=168, idle_age=65534, hard_age=65534, priority=3,arp,dl_vlan=1,arp_tpa=10.0.1.1 actions=drop
 cookie=0x0, duration=64082.449s, table=1, n_packets=2, n_bytes=84, idle_age=63991, priority=3,arp,dl_vlan=2,arp_tpa=10.0.2.1 actions=drop
 cookie=0x0, duration=67599.22s, table=1, n_packets=123, n_bytes=10687, idle_age=1741, hard_age=65534, priority=0 actions=resubmit(,2)

  先匹配table 0,然后匹配table 1,它会把源MAC地址(另一个qr口)改为全局唯一与计算节点绑定的MAC。

  同时,后面的两条table1会丢弃目标ip是interface_distributed接口的ARP和目的MAC是interface_distributed的包,以防止虚机发送给本地IP的包不会被转发到网络中。

  然后继续查询table 2,table 2是vxlan表,如果是广播包就会查询表22,如果是单播包就查询table 20

cookie=0x0, duration=67601.554s, table=2, n_packets=176, n_bytes=16981, idle_age=20810, hard_age=65534, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
 cookie=0x0, duration=67601.406s, table=2, n_packets=92, n_bytes=7876, idle_age=1741, hard_age=65534, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)

  广播MAC地址是FF:FF:FF:FF:FF:FF,组播MAC地址以01-00-5E开头(具体可查看http://book.51cto.com/art/200904/120471.htm),匹配规则满足CIDR。

  ICMP包是单播包,因此会查询表20,由于开启了L2 pop功能,在表20中会事先学习到应该转发到哪个VTEP:

cookie=0x0, duration=64015.308s, table=20, n_packets=0, n_bytes=0, idle_age=64015, priority=2,dl_vlan=2,dl_dst=fa:16:3e:51:99:b8 actions=strip_vlan,set_tunnel:0x3eb,output:4

  (Q2:社区br-tun下面的隧道口是如何与物理口建立联系的?)

  5)进compute node 2 -> br-tun

  在br-tun中,从外面进入的报文将首先匹配以下table0表:

 cookie=0x0, duration=66293.658s, table=0, n_packets=31, n_bytes=3936, idle_age=22651, hard_age=65534, priority=1,in_port=3 actions=resubmit(,4)
 cookie=0x0, duration=69453.368s, table=0, n_packets=103, n_bytes=9360, idle_age=22651, hard_age=65534, priority=1,in_port=1 actions=resubmit(,1)
 cookie=0x0, duration=66292.808s, table=0, n_packets=20, n_bytes=1742, idle_age=3598, hard_age=65534, priority=1,in_port=4 actions=resubmit(,4)
 cookie=0x0, duration=69455.675s, table=0, n_packets=5, n_bytes=390, idle_age=65534, hard_age=65534, priority=0 actions=drop

  在table 4中,会将对应的vni改为本地vlan id,之后查询表9:

 cookie=0x0, duration=65937.871s, table=4, n_packets=32, n_bytes=3653, idle_age=22651, hard_age=65534, priority=1,tun_id=0x3eb actions=mod_vlan_vid:3,resubmit(,9)
 cookie=0x0, duration=66294.732s, table=4, n_packets=19, n_bytes=2025, idle_age=3598, hard_age=65534, priority=1,tun_id=0x3e9 actions=mod_vlan_vid:2,resubmit(,9)
 cookie=0x0, duration=69455.115s, table=4, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop

  在表9中,如果发现包的源地址是全局唯一并与计算节点绑定的MAC地址,就将其转发到br-int:

cookie=0x0, duration=69453.507s, table=9, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=1,dl_src=fa:16:3f:fe:49:e9 actions=output:1
 cookie=0x0, duration=69453.782s, table=9, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=1,dl_src=fa:16:3f:72:3f:a7 actions=output:1
 cookie=0x0, duration=69453.23s, table=9, n_packets=56, n_bytes=6028, idle_age=3598, hard_age=65534, priority=0 actions=resubmit(,10)

  6)br-tun -> br-int

  进入br-int后,在table 0中,如果是全局唯一并与计算节点绑定的MAC地址就查询table 1,否则就正常转发;

  在table 1中,事先设定好了flow,如果目的MAC是发送给VM2,就将源MAC改为Net2的网关MAC地址(qr口)(Q3:修改源MAC的原因)

cookie=0x0, duration=70039.903s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2,in_port=6,dl_src=fa:16:3f:72:3f:a7 actions=resubmit(,1)
 cookie=0x0, duration=70039.627s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2,in_port=6,dl_src=fa:16:3f:fe:49:e9 actions=resubmit(,1)
 cookie=0x0, duration=70040.053s, table=0, n_packets=166, n_bytes=15954, idle_age=4184, hard_age=65534, priority=1 actions=NORMAL
 cookie=0x0, duration=66458.695s, table=1, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=4,dl_vlan=3,dl_dst=fa:16:3e:51:99:b8 actions=strip_vlan,mod_dl_src:fa:16:3e:69:b4:05,output:12
 cookie=0x0, duration=66877.515s, table=1, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=4,dl_vlan=2,dl_dst=fa:16:3e:14:b8:ec actions=strip_vlan,mod_dl_src:fa:16:3e:66:13:af,output:9
 cookie=0x0, duration=66877.369s, table=1, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2,ip,dl_vlan=2,nw_dst=10.0.1.0/24 actions=strip_vlan,mod_dl_src:fa:16:3e:66:13:af,output:9
 cookie=0x0, duration=66458.559s, table=1, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2,ip,dl_vlan=3,nw_dst=10.0.2.0/24 actions=strip_vlan,mod_dl_src:fa:16:3e:69:b4:05,output:12

  7)br-int -> VM2

  至此,VM2就会收到VM1的包了。从通信的过程可以看到,跨网段的东西向流量没有经过网络节点。

  2.2 南北向流量(VM有floating ip)   

  VM1 (local ip:10.0.1.5 , floating ip: 172.24.4.5)ping 8.8.8.8

  1)VM1 (10.0.1.5) -> qr (10.0.1.1)

    与上面一致

  2) qr (10.0.1.1) -> rfp (169.254.31.28) -> fpr (169.254.31.29)

  进入qrouter namespace后:

# ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa ip rule  
0: from all lookup local  
32766: from all lookup main  
32767: from all lookup default  
32768: from 10.0.1.5 lookup 16  
32769: from 10.0.2.3 lookup 16  
167772417: from 10.0.1.1/24 lookup 167772417  
167772417: from 10.0.1.1/24 lookup 167772417  
167772673: from 10.0.2.1/24 lookup 167772673

  在main表中没有合适的路由:

# ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa ip route list table main  
10.0.1.0/24 dev qr-ddbdc784-d7 proto kernel scope link src 10.0.1.1  
10.0.2.0/24 dev qr-001d0ed9-01 proto kernel scope link src 10.0.2.1  
169.254.31.28/31 dev rfp-0fbb351e-a proto kernel scope link src 169.254.31.28

  由于包是从10.0.1.5发来的之后会查看table 16,包会命中这条路由。

# ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa ip route list table 16  
default via 169.254.31.29 dev rfp-0fbb351e-a

  路由之后会通过netfilter的POSTROUTING链中进行SNAT:

# ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa iptables -nvL -t nat
...
Chain neutron-l3-agent-float-snat (1 references)
 pkts bytes target prot opt in out source destination
    0 0 SNAT all -- * * 10.0.2.3 0.0.0.0/0 to:172.24.4.7
    0 0 SNAT all -- * * 10.0.1.5 0.0.0.0/0 to:172.24.4.5
...

  之后就可以看到包会通过rfp-0fbb351e-a发送给169.254.31.29。

  端口rfp-0fbb351e-a和fpr-0fbb351e-a是一对veth pair。在fip namespace中你可以看到这个接口:

  3) fpr (169.254.31.29) -> fg (172.24.4.6)

  到了fip的namespace之后,会查询路由, 在main表里有通往公网的默认路由:

# ip netns exec fip-fbd46644-c70f-4227-a414-862a00cbd1d2 ip route  
default via 172.24.4.1 dev fg-081d537b-06  
169.254.31.28/31 dev fpr-0fbb351e-a proto kernel scope link src 169.254.31.29  
172.24.4.0/24 dev fg-081d537b-06 proto kernel scope link src 172.24.4.6  
172.24.4.5 via 169.254.31.28 dev fpr-0fbb351e-a  
172.24.4.7 via 169.254.31.28 dev fpr-0fbb351e-a

  通过fg-081d537b-06发送到br-ex。这是从虚机发送到公网的过程。(Q4:br-ex上的流表是什么样的?如果没有br-ex,直接走br-int,流表会有什么变化?)

  

  外网 ping VM1 ( floating ip: 172.24.4.5)

  1)fip namespace

  此时fip的namespace会做arp代理:(Q5:arp代理的作用?

# ip netns exec fip-fbd46644-c70f-4227-a414-862a00cbd1d2 sysctl net.ipv4.conf.fg-081d537b-06.proxy_arp  
net.ipv4.conf.fg-081d537b-06.proxy_arp = 1

  可以看到接口的arp代理是打开的,对于floating ip 有以下路由:

# ip netns exec fip-fbd46644-c70f-4227-a414-862a00cbd1d2 ip route  
...
172.24.4.5 via 169.254.31.28 dev fpr-0fbb351e-a  
172.24.4.7 via 169.254.31.28 dev fpr-0fbb351e-a
...

  ARP会去通过VETH Pair到IR(Inter Router)的namespace中去查询,在IR中可以看到,接口rfp-0fbb351e-a配置了floating ip:

# ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa ip addr 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default  
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 
    inet 127.0.0.1/8 scope host lo 
       valid_lft forever preferred_lft forever 
    inet6 ::1/128 scope host  
       valid_lft forever preferred_lft forever 
2: rfp-0fbb351e-a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether ea:5c:56:9a:36:9c brd ff:ff:ff:ff:ff:ff
    inet 169.254.31.28/31 scope global rfp-0fbb351e-a
       valid_lft forever preferred_lft forever
    inet 172.24.4.5/32 brd 172.24.4.5 scope global rfp-0fbb351e-a
       valid_lft forever preferred_lft forever
    inet 172.24.4.7/32 brd 172.24.4.7 scope global rfp-0fbb351e-a
       valid_lft forever preferred_lft forever
    inet6 fe80::e85c:56ff:fe9a:369c/64 scope link 
       valid_lft forever preferred_lft forever 
17: qr-ddbdc784-d7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default  
    link/ether fa:16:3e:66:13:af brd ff:ff:ff:ff:ff:ff 
    inet 10.0.1.1/24 brd 10.0.1.255 scope global qr-ddbdc784-d7 
       valid_lft forever preferred_lft forever 
    inet6 fe80::f816:3eff:fe66:13af/64 scope link  
       valid_lft forever preferred_lft forever 
19: qr-001d0ed9-01: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default  
    link/ether fa:16:3e:69:b4:05 brd ff:ff:ff:ff:ff:ff 
    inet 10.0.2.1/24 brd 10.0.2.255 scope global qr-001d0ed9-01 
       valid_lft forever preferred_lft forever 
    inet6 fe80::f816:3eff:fe69:b405/64 scope link  
       valid_lft forever preferred_lft forever

  因此fip的namespace会对这个floating ip进行ARP回应。

  外部发起目标地址为floating ip的请求后,fip会将其转发到IR中,IR的RPOROUTING链中规则如下:

# ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa iptables -nvL -t nat
...
Chain neutron-l3-agent-PREROUTING (1 references)
 pkts bytes target prot opt in out source destination
    0 0 REDIRECT tcp -- * * 0.0.0.0/0 169.254.169.254 tcp dpt:80 redir ports 9697
    0 0 DNAT all -- * * 0.0.0.0/0 172.24.4.7 to:10.0.2.3
    0 0 DNAT all -- * * 0.0.0.0/0 172.24.4.5 to:10.0.1.5
...

  这条DNAT规则会将floating ip地址转换为内部地址,之后进行路由查询:

# ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa ip route  
10.0.1.0/24 dev qr-ddbdc784-d7 proto kernel scope link src 10.0.1.1  
10.0.2.0/24 dev qr-001d0ed9-01 proto kernel scope link src 10.0.2.1  
169.254.31.28/31 dev rfp-0fbb351e-a proto kernel scope link src 169.254.31.28

  目的地址是10.0.1.0/24网段的,因此会从qr-ddbdc784-d7转发出去。之后就会转发到br-int再到虚机。

 

  2.3 南北向流量(VM没有floating ip)

  在虚机没有floating ip的情况下,从虚机发出的包会首先到IR,IR中查询路由:

# ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa ip rule  
0: from all lookup local  
32766: from all lookup main  
32767: from all lookup default  
32768: from 10.0.1.5 lookup 16  
32769: from 10.0.2.3 lookup 16  
167772417: from 10.0.1.1/24 lookup 167772417   
167772673: from 10.0.2.1/24 lookup 167772673

  会先查询main表,之后查询167772417表。(Q7:不会匹配table 16?) 

# ip netns exec qrouter-0fbb351e-a65b-4790-a409-8fb219ce16aa ip route list table 167772417  
default via 10.0.1.6 dev qr-ddbdc784-d7

  这个表会将其转发给10.0.1.6,而这个IP就是在network node上的router_centralized_snat接口。

  在network node的snat namespace中,我们可以看到这个接口。

$ sudo ip netns exec snat-0fbb351e-a65b-4790-a409-8fb219ce16aa iptables -nvL -t nat
...
Chain neutron-l3-agent-snat (1 references)
 pkts bytes target prot opt in out source destination
    0 0 SNAT all -- * * 10.0.1.0/24 0.0.0.0/0 to:172.24.4.4
    0 0 SNAT all -- * * 10.0.2.0/24 0.0.0.0/0 to:172.24.4.4
...

  这里就和以前的L3类似,会将没有floating ip的包SNAT成一个172.24.4.4(DVR的网关臂)。这个过程是和以前L3类似的,不再累述。

  参考:http://www.sxt.cn/u/756/blog/3168

3. QA

  (未完)

 

DVR分布式路由

标签:

原文地址:http://www.cnblogs.com/xingyun/p/4876083.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!