openstack neutron DVR internal with floating IP

| 分类 Openstack  | 标签 neutron 

Add floating IP for VM

[root@controller ~]# neutron l3-agent-list-hosting-router demo-router
+--------------------------------------+----------+----------------+-------+----------+
| id                                   | host     | admin_state_up | alive | ha_state |
+--------------------------------------+----------+----------------+-------+----------+
| 5dc96667-7f5f-4e3d-85aa-22759c7a4c5a | compute1 | True           | :-)   |          |
| 77c78491-d4e6-491f-8ac6-e6348fbc87c9 | network  | True           | :-)   |          |
+--------------------------------------+----------+----------------+-------+----------+

Create a floating IP address on the external network

[root@controller ~]# neutron floatingip-create ext-net
Created a new floatingip:
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| fixed_ip_address    |                                      |
| floating_ip_address | 203.0.113.11                         |
| floating_network_id | 2f00f5f1-ad6f-4113-b550-df9f33afae49 |
| id                  | 8767bb49-dad7-4118-b8cc-309225489242 |
| port_id             |                                      |
| router_id           |                                      |
| status              | DOWN                                 |
| tenant_id           | 0b48811a7e614b1ba752a6544d4eba02     |
+---------------------+--------------------------------------+

Associate the floating IP address with the instance

[root@controller ~]# nova floating-ip-associate demo-vm1 203.0.113.11

[root@controller ~]# nova list
+--------------------------------------+----------+--------+------------+-------------+-------------------------------------+
| ID                                   | Name     | Status | Task State | Power State | Networks                            |
+--------------------------------------+----------+--------+------------+-------------+-------------------------------------+
| 5dd45381-ee24-4d37-b8e0-2a985e744ae2 | demo-vm1 | ACTIVE | -          | Running     | demo-net=192.168.10.4, 203.0.113.11 |
+--------------------------------------+----------+--------+------------+-------------+-------------------------------------+

compute node’s network

OVS ports

[root@compute1 ~]# ovs-vsctl show
0ebd643a-fbf9-482c-8488-9292ad146e4a
    Bridge br-ex
        Port "fg-50b88d31-14"
            Interface "fg-50b88d31-14"
                type: internal
        Port "eth2"
            Interface "eth2"
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port br-ex
            Interface br-ex
                type: internal
    Bridge br-int
        fail_mode: secure
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port br-int
            Interface br-int
                type: internal
        Port "qvofa8befcd-cd"
            tag: 1
            Interface "qvofa8befcd-cd"
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "qr-e97ad7a4-64"
            tag: 1
            Interface "qr-e97ad7a4-64"
                type: internal
    Bridge br-tun
        fail_mode: secure
        Port br-tun
            Interface br-tun
                type: internal
        Port "vxlan-ac122a33"
            Interface "vxlan-ac122a33"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="172.18.42.52", out_key=flow, remote_ip="172.18.42.51"}
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    ovs_version: "2.4.0"

qr-XXX netnamespace

[root@compute1 ~]# ip netns exec qrouter-dd6481d3-c3c6-4cf8-877c-877c94b2edd3 ip a   
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: rfp-dd6481d3-c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 82:de:1b:13:37:d8 brd ff:ff:ff:ff:ff:ff
    inet 169.254.31.28/31 scope global rfp-dd6481d3-c
       valid_lft forever preferred_lft forever
    inet 203.0.113.11/32 brd 203.0.113.11 scope global rfp-dd6481d3-c
       valid_lft forever preferred_lft forever
    inet6 fe80::80de:1bff:fe13:37d8/64 scope link 
       valid_lft forever preferred_lft forever
14: qr-e97ad7a4-64: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:ec:a7:5e brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.1/24 brd 192.168.10.255 scope global qr-e97ad7a4-64
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:feec:a75e/64 scope link 
       valid_lft forever preferred_lft forever

[root@compute1 ~]# ip netns exec qrouter-dd6481d3-c3c6-4cf8-877c-877c94b2edd3 ip rule
0:      from all lookup local 
32766:  from all lookup main 
32767:  from all lookup default 
57481:  from 192.168.10.4 lookup 16 
3232238081:     from 192.168.10.1/24 lookup 3232238081 

[root@compute1 ~]# ip netns exec qrouter-dd6481d3-c3c6-4cf8-877c-877c94b2edd3 ip route list table 16
default via 169.254.31.29 dev rfp-dd6481d3-c 

[root@compute1 ~]# ip netns exec qrouter-dd6481d3-c3c6-4cf8-877c-877c94b2edd3 iptables -t nat -S    
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-OUTPUT -d 203.0.113.11/32 -j DNAT --to-destination 192.168.10.4
-A neutron-l3-agent-POSTROUTING ! -i rfp-dd6481d3-c ! -o rfp-dd6481d3-c -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 203.0.113.11/32 -j DNAT --to-destination 192.168.10.4
-A neutron-l3-agent-float-snat -s 192.168.10.4/32 -j SNAT --to-source 203.0.113.11
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat

可以看到compute node的qroute增加了一条rule和route table。即所有来自demo-vm1(192.168.10.4)都会从rfp-dd6481d3-c(169.254.31.28)发往下一跳169.254.31.29,并且在发送之前,完成SNAT转换:

-A neutron-l3-agent-float-snat -s 192.168.10.4/32 -j SNAT --to-source 203.0.113.11

实际上,rfp-dd6481d3-c是一个veth设备,与fip netnamespace的fpr-dd6481d3-c(169.254.31.29)相连。

fip netnamespace

[root@compute1 ~]# ip netns
fip-2f00f5f1-ad6f-4113-b550-df9f33afae49
qrouter-dd6481d3-c3c6-4cf8-877c-877c94b2edd3

[root@compute1 ~]# ip netns exec fip-2f00f5f1-ad6f-4113-b550-df9f33afae49 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: fpr-dd6481d3-c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether e6:b2:b3:96:c9:6c brd ff:ff:ff:ff:ff:ff
    inet 169.254.31.29/31 scope global fpr-dd6481d3-c
       valid_lft forever preferred_lft forever
    inet6 fe80::e4b2:b3ff:fe96:c96c/64 scope link 
       valid_lft forever preferred_lft forever
15: fg-50b88d31-14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:a5:23:8f brd ff:ff:ff:ff:ff:ff
    inet 203.0.113.12/24 brd 203.0.113.255 scope global fg-50b88d31-14
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fea5:238f/64 scope link 
       valid_lft forever preferred_lft forever

[root@compute1 ~]# ip netns exec fip-2f00f5f1-ad6f-4113-b550-df9f33afae49 ip route show
default via 203.0.113.1 dev fg-50b88d31-14 
169.254.31.28/31 dev fpr-dd6481d3-c  proto kernel  scope link  src 169.254.31.29 
203.0.113.0/24 dev fg-50b88d31-14  proto kernel  scope link  src 203.0.113.12 
203.0.113.11 via 169.254.31.28 dev fpr-dd6481d3-c 

当数据包(发往203.0.113.x)从VM传到fpr-dd6481d3-c(169.254.31.29)后,匹配第3条路由规则,从fg-50b88d31-14送出,fg-50b88d31-14是br-ex的port。

North/south for instances with a floating IP address

VM -> external

整体流程如下:

$ traceroute 203.0.113.250
traceroute to 203.0.113.250 (203.0.113.250), 30 hops max, 46 byte packets
 1  192.168.10.1 (192.168.10.1)  0.164 ms  0.207 ms  0.209 ms
 2  169.254.31.29 (169.254.31.29)  0.533 ms  0.445 ms  0.356 ms
 3  203.0.113.250 (203.0.113.250)  0.415 ms  0.374 ms  0.381 ms

external -> VM

[root@controller ~]# nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range  | Source Group |
+-------------+-----------+---------+-----------+--------------+
| icmp        | -1        | -1      | 0.0.0.0/0 |              |
+-------------+-----------+---------+-----------+--------------+
[root@controller ~]# nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range  | Source Group |
+-------------+-----------+---------+-----------+--------------+
| tcp         | 22        | 22      | 0.0.0.0/0 |              |
+-------------+-----------+---------+-----------+--------------+

[root@external ~]# ssh cirros@203.0.113.11
cirros@203.0.113.11's password: 
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast qlen 1000
    link/ether fa:16:3e:94:5d:0c brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.4/24 brd 192.168.10.255 scope global eth0
    inet6 fe80::f816:3eff:fe94:5d0c/64 scope link 
       valid_lft forever preferred_lft forever

当数据包到达fg-50b88d31-14(203.0.113.12/24)时,匹配最后一条路由规则:

203.0.113.11 via 169.254.31.28 dev fpr-dd6481d3-c 

传送到fip netnamespace的rfp-dd6481d3-c,然后匹配DNAT规则:

-A neutron-l3-agent-PREROUTING -d 203.0.113.11/32 -j DNAT --to-destination 192.168.10.4

然后匹配qr netnamespace的下面的路由规则:

[root@compute1 ~]# ip netns exec qrouter-dd6481d3-c3c6-4cf8-877c-877c94b2edd3 ip route show     
169.254.31.28/31 dev rfp-dd6481d3-c  proto kernel  scope link  src 169.254.31.28 
192.168.10.0/24 dev qr-e97ad7a4-64  proto kernel  scope link  src 192.168.10.1 

然后从qr-e97ad7a4-64送出。最后经过br-int送给VM。

DVR future

  • VPNaaS support for DVR
  • Full migration support for DVR routers.
  • HA for Service Node
  • IPv6 Support
  • VLAN Support
  • L3 Agent Refactor
  • Distributed DHCP
  • ** Performance tuning **
  • ** Distributed SNAT **

总结

DVR实现的网络比较复杂,VM与external之间的链路也很长,大量的veth设备带来的网络性能和延迟都会是一个问题。

相关资料


上一篇     下一篇