Documentation/networking/vrf.txt - kernel/quantenna - Git at Google

 Virtual Routing and Forwarding (VRF)
 ====================================
 The VRF device combined with ip rules provides the ability to create virtual
 routing and forwarding domains (aka VRFs, VRF-lite to be specific) in the
 Linux network stack. One use case is the multi-tenancy problem where each
 tenant has their own unique routing tables and in the very least need
 different default gateways.

 Processes can be "VRF aware" by binding a socket to the VRF device. Packets
 through the socket then use the routing table associated with the VRF
 device. An important feature of the VRF device implementation is that it
 impacts only Layer 3 and above so L2 tools (e.g., LLDP) are not affected
 (ie., they do not need to be run in each VRF). The design also allows
 the use of higher priority ip rules (Policy Based Routing, PBR) to take
 precedence over the VRF device rules directing specific traffic as desired.

 In addition, VRF devices allow VRFs to be nested within namespaces. For
 example network namespaces provide separation of network interfaces at L1
 (Layer 1 separation), VLANs on the interfaces within a namespace provide
 L2 separation and then VRF devices provide L3 separation.

 Design
 ------
 A VRF device is created with an associated route table. Network interfaces
 are then enslaved to a VRF device:

          +-----------------------------+
          |           vrf-blue          |  ===> route table 10
          +-----------------------------+
             |        |            |
          +------+ +------+     +-------------+
          | eth1 | | eth2 | ... |    bond1    |
          +------+ +------+     +-------------+
                                   |       |
                               +------+ +------+
                               | eth8 | | eth9 |
                               +------+ +------+

 Packets received on an enslaved device and are switched to the VRF device
 using an rx_handler which gives the impression that packets flow through
 the VRF device. Similarly on egress routing rules are used to send packets
 to the VRF device driver before getting sent out the actual interface. This
 allows tcpdump on a VRF device to capture all packets into and out of the
 VRF as a whole.[1] Similarly, netfilter [2] and tc rules can be applied
 using the VRF device to specify rules that apply to the VRF domain as a whole.

 [1] Packets in the forwarded state do not flow through the device, so those
     packets are not seen by tcpdump. Will revisit this limitation in a
     future release.

 [2] Iptables on ingress is limited to NF_INET_PRE_ROUTING only with skb->dev
     set to real ingress device and egress is limited to NF_INET_POST_ROUTING.
     Will revisit this limitation in a future release.


 Setup
 -----
 1. VRF device is created with an association to a FIB table.
    e.g, ip link add vrf-blue type vrf table 10
         ip link set dev vrf-blue up

 2. Rules are added that send lookups to the associated FIB table when the
    iif or oif is the VRF device. e.g.,
        ip ru add oif vrf-blue table 10
        ip ru add iif vrf-blue table 10

    Set the default route for the table (and hence default route for the VRF).
    e.g, ip route add table 10 prohibit default

 3. Enslave L3 interfaces to a VRF device.
    e.g,  ip link set dev eth1 master vrf-blue

    Local and connected routes for enslaved devices are automatically moved to
    the table associated with VRF device. Any additional routes depending on
    the enslaved device will need to be reinserted following the enslavement.

 4. Additional VRF routes are added to associated table.
    e.g., ip route add table 10 ...


 Applications
 ------------
 Applications that are to work within a VRF need to bind their socket to the
 VRF device:

     setsockopt(sd, SOL_SOCKET, SO_BINDTODEVICE, dev, strlen(dev)+1);

 or to specify the output device using cmsg and IP_PKTINFO.


 Limitations
 -----------
 Index of original ingress interface is not available via cmsg. Will address
 soon.

 ################################################################################

 Using iproute2 for VRFs
 =======================
 VRF devices do *not* have to start with 'vrf-'. That is a convention used here
 for emphasis of the device type, similar to use of 'br' in bridge names.

 1. Create a VRF

    To instantiate a VRF device and associate it with a table:
        $ ip link add dev NAME type vrf table ID

    Remember to add the ip rules as well:
        $ ip ru add oif NAME table 10
        $ ip ru add iif NAME table 10
        $ ip -6 ru add oif NAME table 10
        $ ip -6 ru add iif NAME table 10

    Without the rules route lookups are not directed to the table.

    For example:
    $ ip link add dev vrf-blue type vrf table 10
    $ ip ru add pref 200 oif vrf-blue table 10
    $ ip ru add pref 200 iif vrf-blue table 10
    $ ip -6 ru add pref 200 oif vrf-blue table 10
    $ ip -6 ru add pref 200 iif vrf-blue table 10


 2. List VRFs

    To list VRFs that have been created:
        $ ip [-d] link show type vrf
          NOTE: The -d option is needed to show the table id

    For example:
    $ ip -d link show type vrf
    11: vrf-mgmt: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
        link/ether 72:b3:ba:91:e2:24 brd ff:ff:ff:ff:ff:ff promiscuity 0
        vrf table 1 addrgenmode eui64
    12: vrf-red: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
        link/ether b6:6f:6e:f6:da:73 brd ff:ff:ff:ff:ff:ff promiscuity 0
        vrf table 10 addrgenmode eui64
    13: vrf-blue: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
        link/ether 36:62:e8:7d:bb:8c brd ff:ff:ff:ff:ff:ff promiscuity 0
        vrf table 66 addrgenmode eui64
    14: vrf-green: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
        link/ether e6:28:b8:63:70:bb brd ff:ff:ff:ff:ff:ff promiscuity 0
        vrf table 81 addrgenmode eui64


    Or in brief output:

    $ ip -br link show type vrf
    vrf-mgmt         UP             72:b3:ba:91:e2:24 <NOARP,MASTER,UP,LOWER_UP>
    vrf-red          UP             b6:6f:6e:f6:da:73 <NOARP,MASTER,UP,LOWER_UP>
    vrf-blue         UP             36:62:e8:7d:bb:8c <NOARP,MASTER,UP,LOWER_UP>
    vrf-green        UP             e6:28:b8:63:70:bb <NOARP,MASTER,UP,LOWER_UP>


 3. Assign a Network Interface to a VRF

    Network interfaces are assigned to a VRF by enslaving the netdevice to a
    VRF device:
        $ ip link set dev NAME master VRF-NAME

    On enslavement connected and local routes are automatically moved to the
    table associated with the VRF device.

    For example:
    $ ip link set dev eth0 master vrf-mgmt


 4. Show Devices Assigned to a VRF

    To show devices that have been assigned to a specific VRF add the master
    option to the ip command:
        $ ip link show master VRF-NAME

    For example:
    $ ip link show master vrf-red
    3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vrf-red state UP mode DEFAULT group default qlen 1000
        link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
    4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vrf-red state UP mode DEFAULT group default qlen 1000
        link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
    7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master vrf-red state DOWN mode DEFAULT group default qlen 1000
        link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff


    Or using the brief output:
    $ ip -br link show master vrf-red
    eth1             UP             02:00:00:00:02:02 <BROADCAST,MULTICAST,UP,LOWER_UP>
    eth2             UP             02:00:00:00:02:03 <BROADCAST,MULTICAST,UP,LOWER_UP>
    eth5             DOWN           02:00:00:00:02:06 <BROADCAST,MULTICAST>


 5. Show Neighbor Entries for a VRF

    To list neighbor entries associated with devices enslaved to a VRF device
    add the master option to the ip command:
        $ ip [-6] neigh show master VRF-NAME

    For example:
    $  ip neigh show master vrf-red
    10.2.1.254 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
    10.2.2.254 dev eth2 lladdr 5e:54:01:6a:ee:80 REACHABLE

     $ ip -6 neigh show master vrf-red
     2002:1::64 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE


 6. Show Addresses for a VRF

    To show addresses for interfaces associated with a VRF add the master
    option to the ip command:
        $ ip addr show master VRF-NAME

    For example:
    $ ip addr show master vrf-red
    3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vrf-red state UP group default qlen 1000
        link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
        inet 10.2.1.2/24 brd 10.2.1.255 scope global eth1
           valid_lft forever preferred_lft forever
        inet6 2002:1::2/120 scope global
           valid_lft forever preferred_lft forever
        inet6 fe80::ff:fe00:202/64 scope link
           valid_lft forever preferred_lft forever
    4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vrf-red state UP group default qlen 1000
        link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
        inet 10.2.2.2/24 brd 10.2.2.255 scope global eth2
           valid_lft forever preferred_lft forever
        inet6 2002:2::2/120 scope global
           valid_lft forever preferred_lft forever
        inet6 fe80::ff:fe00:203/64 scope link
           valid_lft forever preferred_lft forever
    7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master vrf-red state DOWN group default qlen 1000
        link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff

    Or in brief format:
    $ ip -br addr show master vrf-red
    eth1             UP             10.2.1.2/24 2002:1::2/120 fe80::ff:fe00:202/64
    eth2             UP             10.2.2.2/24 2002:2::2/120 fe80::ff:fe00:203/64
    eth5             DOWN


 7. Show Routes for a VRF

    To show routes for a VRF use the ip command to display the table associated
    with the VRF device:
        $ ip [-6] route show table ID

    For example:
    $ ip route show table vrf-red
    prohibit default
    broadcast 10.2.1.0 dev eth1  proto kernel  scope link  src 10.2.1.2
    10.2.1.0/24 dev eth1  proto kernel  scope link  src 10.2.1.2
    local 10.2.1.2 dev eth1  proto kernel  scope host  src 10.2.1.2
    broadcast 10.2.1.255 dev eth1  proto kernel  scope link  src 10.2.1.2
    broadcast 10.2.2.0 dev eth2  proto kernel  scope link  src 10.2.2.2
    10.2.2.0/24 dev eth2  proto kernel  scope link  src 10.2.2.2
    local 10.2.2.2 dev eth2  proto kernel  scope host  src 10.2.2.2
    broadcast 10.2.2.255 dev eth2  proto kernel  scope link  src 10.2.2.2

    $ ip -6 route show table vrf-red
    local 2002:1:: dev lo  proto none  metric 0  pref medium
    local 2002:1::2 dev lo  proto none  metric 0  pref medium
    2002:1::/120 dev eth1  proto kernel  metric 256  pref medium
    local 2002:2:: dev lo  proto none  metric 0  pref medium
    local 2002:2::2 dev lo  proto none  metric 0  pref medium
    2002:2::/120 dev eth2  proto kernel  metric 256  pref medium
    local fe80:: dev lo  proto none  metric 0  pref medium
    local fe80:: dev lo  proto none  metric 0  pref medium
    local fe80::ff:fe00:202 dev lo  proto none  metric 0  pref medium
    local fe80::ff:fe00:203 dev lo  proto none  metric 0  pref medium
    fe80::/64 dev eth1  proto kernel  metric 256  pref medium
    fe80::/64 dev eth2  proto kernel  metric 256  pref medium
    ff00::/8 dev vrf-red  metric 256  pref medium
    ff00::/8 dev eth1  metric 256  pref medium
    ff00::/8 dev eth2  metric 256  pref medium


 8. Route Lookup for a VRF

    A test route lookup can be done for a VRF by adding the oif option to ip:
        $ ip [-6] route get oif VRF-NAME ADDRESS

    For example:
    $ ip route get 10.2.1.40 oif vrf-red
    10.2.1.40 dev eth1  table vrf-red  src 10.2.1.2
        cache

    $ ip -6 route get 2002:1::32 oif vrf-red
    2002:1::32 from :: dev eth1  table vrf-red  proto kernel  src 2002:1::2  metric 256  pref medium


 9. Removing Network Interface from a VRF

    Network interfaces are removed from a VRF by breaking the enslavement to
    the VRF device:
        $ ip link set dev NAME nomaster

    Connected routes are moved back to the default table and local entries are
    moved to the local table.

    For example:
    $ ip link set dev eth0 nomaster

 --------------------------------------------------------------------------------

 Commands used in this example:

 cat >> /etc/iproute2/rt_tables <<EOF
 1  vrf-mgmt
 10 vrf-red
 66 vrf-blue
 81 vrf-green
 EOF

 function vrf_create
 {
     VRF=$1
     TBID=$2
     # create VRF device
     ip link add vrf-${VRF} type vrf table ${TBID}

     # add rules that direct lookups to vrf table
     ip ru add pref 200 oif vrf-${VRF} table ${TBID}
     ip ru add pref 200 iif vrf-${VRF} table ${TBID}
     ip -6 ru add pref 200 oif vrf-${VRF} table ${TBID}
     ip -6 ru add pref 200 iif vrf-${VRF} table ${TBID}

     if [ "${VRF}" != "mgmt" ]; then
         ip route add table ${TBID} prohibit default
     fi
     ip link set dev vrf-${VRF} up
     ip link set dev vrf-${VRF} state up
 }

 vrf_create mgmt 1
 ip link set dev eth0 master vrf-mgmt

 vrf_create red 10
 ip link set dev eth1 master vrf-red
 ip link set dev eth2 master vrf-red
 ip link set dev eth5 master vrf-red

 vrf_create blue 66
 ip link set dev eth3 master vrf-blue

 vrf_create green 81
 ip link set dev eth4 master vrf-green


 Interface addresses from /etc/network/interfaces:
 auto eth0
 iface eth0 inet static
       address 10.0.0.2
       netmask 255.255.255.0
       gateway 10.0.0.254

 iface eth0 inet6 static
       address 2000:1::2
       netmask 120

 auto eth1
 iface eth1 inet static
       address 10.2.1.2
       netmask 255.255.255.0

 iface eth1 inet6 static
       address 2002:1::2
       netmask 120

 auto eth2
 iface eth2 inet static
       address 10.2.2.2
       netmask 255.255.255.0

 iface eth2 inet6 static
       address 2002:2::2
       netmask 120

 auto eth3
 iface eth3 inet static
       address 10.2.3.2
       netmask 255.255.255.0

 iface eth3 inet6 static
       address 2002:3::2
       netmask 120

 auto eth4
 iface eth4 inet static
       address 10.2.4.2
       netmask 255.255.255.0

 iface eth4 inet6 static
       address 2002:4::2
       netmask 120
	Virtual Routing and Forwarding (VRF)
	====================================
	The VRF device combined with ip rules provides the ability to create virtual
	routing and forwarding domains (aka VRFs, VRF-lite to be specific) in the
	Linux network stack. One use case is the multi-tenancy problem where each
	tenant has their own unique routing tables and in the very least need
	different default gateways.

	Processes can be "VRF aware" by binding a socket to the VRF device. Packets
	through the socket then use the routing table associated with the VRF
	device. An important feature of the VRF device implementation is that it
	impacts only Layer 3 and above so L2 tools (e.g., LLDP) are not affected
	(ie., they do not need to be run in each VRF). The design also allows
	the use of higher priority ip rules (Policy Based Routing, PBR) to take
	precedence over the VRF device rules directing specific traffic as desired.

	In addition, VRF devices allow VRFs to be nested within namespaces. For
	example network namespaces provide separation of network interfaces at L1
	(Layer 1 separation), VLANs on the interfaces within a namespace provide
	L2 separation and then VRF devices provide L3 separation.

	Design
	------
	A VRF device is created with an associated route table. Network interfaces
	are then enslaved to a VRF device:

	+-----------------------------+
	\| vrf-blue \| ===> route table 10
	+-----------------------------+
	\| \| \|
	+------+ +------+ +-------------+
	\| eth1 \| \| eth2 \| ... \| bond1 \|
	+------+ +------+ +-------------+
	\| \|
	+------+ +------+
	\| eth8 \| \| eth9 \|
	+------+ +------+

	Packets received on an enslaved device and are switched to the VRF device
	using an rx_handler which gives the impression that packets flow through
	the VRF device. Similarly on egress routing rules are used to send packets
	to the VRF device driver before getting sent out the actual interface. This
	allows tcpdump on a VRF device to capture all packets into and out of the
	VRF as a whole.[1] Similarly, netfilter [2] and tc rules can be applied
	using the VRF device to specify rules that apply to the VRF domain as a whole.

	[1] Packets in the forwarded state do not flow through the device, so those
	packets are not seen by tcpdump. Will revisit this limitation in a
	future release.

	[2] Iptables on ingress is limited to NF_INET_PRE_ROUTING only with skb->dev
	set to real ingress device and egress is limited to NF_INET_POST_ROUTING.
	Will revisit this limitation in a future release.


	Setup
	-----
	1. VRF device is created with an association to a FIB table.
	e.g, ip link add vrf-blue type vrf table 10
	ip link set dev vrf-blue up

	2. Rules are added that send lookups to the associated FIB table when the
	iif or oif is the VRF device. e.g.,
	ip ru add oif vrf-blue table 10
	ip ru add iif vrf-blue table 10

	Set the default route for the table (and hence default route for the VRF).
	e.g, ip route add table 10 prohibit default

	3. Enslave L3 interfaces to a VRF device.
	e.g, ip link set dev eth1 master vrf-blue

	Local and connected routes for enslaved devices are automatically moved to
	the table associated with VRF device. Any additional routes depending on
	the enslaved device will need to be reinserted following the enslavement.

	4. Additional VRF routes are added to associated table.
	e.g., ip route add table 10 ...


	Applications
	------------
	Applications that are to work within a VRF need to bind their socket to the
	VRF device:

	setsockopt(sd, SOL_SOCKET, SO_BINDTODEVICE, dev, strlen(dev)+1);

	or to specify the output device using cmsg and IP_PKTINFO.


	Limitations
	-----------
	Index of original ingress interface is not available via cmsg. Will address
	soon.

	################################################################################

	Using iproute2 for VRFs
	=======================
	VRF devices do not have to start with 'vrf-'. That is a convention used here
	for emphasis of the device type, similar to use of 'br' in bridge names.

	1. Create a VRF

	To instantiate a VRF device and associate it with a table:
	$ ip link add dev NAME type vrf table ID

	Remember to add the ip rules as well:
	$ ip ru add oif NAME table 10
	$ ip ru add iif NAME table 10
	$ ip -6 ru add oif NAME table 10
	$ ip -6 ru add iif NAME table 10

	Without the rules route lookups are not directed to the table.

	For example:
	$ ip link add dev vrf-blue type vrf table 10
	$ ip ru add pref 200 oif vrf-blue table 10
	$ ip ru add pref 200 iif vrf-blue table 10
	$ ip -6 ru add pref 200 oif vrf-blue table 10
	$ ip -6 ru add pref 200 iif vrf-blue table 10


	2. List VRFs

	To list VRFs that have been created:
	$ ip [-d] link show type vrf
	NOTE: The -d option is needed to show the table id

	For example:
	$ ip -d link show type vrf
	11: vrf-mgmt: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
	link/ether 72:b3:ba:91:e2:24 brd ff:ff:ff:ff:ff:ff promiscuity 0
	vrf table 1 addrgenmode eui64
	12: vrf-red: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
	link/ether b6:6f:6e:f6:da:73 brd ff:ff:ff:ff:ff:ff promiscuity 0
	vrf table 10 addrgenmode eui64
	13: vrf-blue: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
	link/ether 36:62:e8:7d:bb:8c brd ff:ff:ff:ff:ff:ff promiscuity 0
	vrf table 66 addrgenmode eui64
	14: vrf-green: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
	link/ether e6:28:b8:63:70:bb brd ff:ff:ff:ff:ff:ff promiscuity 0
	vrf table 81 addrgenmode eui64


	Or in brief output:

	$ ip -br link show type vrf
	vrf-mgmt UP 72:b3:ba:91:e2:24 <NOARP,MASTER,UP,LOWER_UP>
	vrf-red UP b6:6f:6e:f6:da:73 <NOARP,MASTER,UP,LOWER_UP>
	vrf-blue UP 36:62:e8:7d:bb:8c <NOARP,MASTER,UP,LOWER_UP>
	vrf-green UP e6:28:b8:63:70:bb <NOARP,MASTER,UP,LOWER_UP>


	3. Assign a Network Interface to a VRF

	Network interfaces are assigned to a VRF by enslaving the netdevice to a
	VRF device:
	$ ip link set dev NAME master VRF-NAME

	On enslavement connected and local routes are automatically moved to the
	table associated with the VRF device.

	For example:
	$ ip link set dev eth0 master vrf-mgmt


	4. Show Devices Assigned to a VRF

	To show devices that have been assigned to a specific VRF add the master
	option to the ip command:
	$ ip link show master VRF-NAME

	For example:
	$ ip link show master vrf-red
	3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vrf-red state UP mode DEFAULT group default qlen 1000
	link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
	4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vrf-red state UP mode DEFAULT group default qlen 1000
	link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
	7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master vrf-red state DOWN mode DEFAULT group default qlen 1000
	link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff


	Or using the brief output:
	$ ip -br link show master vrf-red
	eth1 UP 02:00:00:00:02:02 <BROADCAST,MULTICAST,UP,LOWER_UP>
	eth2 UP 02:00:00:00:02:03 <BROADCAST,MULTICAST,UP,LOWER_UP>
	eth5 DOWN 02:00:00:00:02:06 <BROADCAST,MULTICAST>


	5. Show Neighbor Entries for a VRF

	To list neighbor entries associated with devices enslaved to a VRF device
	add the master option to the ip command:
	$ ip [-6] neigh show master VRF-NAME

	For example:
	$ ip neigh show master vrf-red
	10.2.1.254 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
	10.2.2.254 dev eth2 lladdr 5e:54:01:6a:ee:80 REACHABLE

	$ ip -6 neigh show master vrf-red
	2002:1::64 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE


	6. Show Addresses for a VRF

	To show addresses for interfaces associated with a VRF add the master
	option to the ip command:
	$ ip addr show master VRF-NAME

	For example:
	$ ip addr show master vrf-red
	3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vrf-red state UP group default qlen 1000
	link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
	inet 10.2.1.2/24 brd 10.2.1.255 scope global eth1
	valid_lft forever preferred_lft forever
	inet6 2002:1::2/120 scope global
	valid_lft forever preferred_lft forever
	inet6 fe80::ff:fe00:202/64 scope link
	valid_lft forever preferred_lft forever
	4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vrf-red state UP group default qlen 1000
	link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
	inet 10.2.2.2/24 brd 10.2.2.255 scope global eth2
	valid_lft forever preferred_lft forever
	inet6 2002:2::2/120 scope global
	valid_lft forever preferred_lft forever
	inet6 fe80::ff:fe00:203/64 scope link
	valid_lft forever preferred_lft forever
	7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master vrf-red state DOWN group default qlen 1000
	link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff

	Or in brief format:
	$ ip -br addr show master vrf-red
	eth1 UP 10.2.1.2/24 2002:1::2/120 fe80::ff:fe00:202/64
	eth2 UP 10.2.2.2/24 2002:2::2/120 fe80::ff:fe00:203/64
	eth5 DOWN


	7. Show Routes for a VRF

	To show routes for a VRF use the ip command to display the table associated
	with the VRF device:
	$ ip [-6] route show table ID

	For example:
	$ ip route show table vrf-red
	prohibit default
	broadcast 10.2.1.0 dev eth1 proto kernel scope link src 10.2.1.2
	10.2.1.0/24 dev eth1 proto kernel scope link src 10.2.1.2
	local 10.2.1.2 dev eth1 proto kernel scope host src 10.2.1.2
	broadcast 10.2.1.255 dev eth1 proto kernel scope link src 10.2.1.2
	broadcast 10.2.2.0 dev eth2 proto kernel scope link src 10.2.2.2
	10.2.2.0/24 dev eth2 proto kernel scope link src 10.2.2.2
	local 10.2.2.2 dev eth2 proto kernel scope host src 10.2.2.2
	broadcast 10.2.2.255 dev eth2 proto kernel scope link src 10.2.2.2

	$ ip -6 route show table vrf-red
	local 2002:1:: dev lo proto none metric 0 pref medium
	local 2002:1::2 dev lo proto none metric 0 pref medium
	2002:1::/120 dev eth1 proto kernel metric 256 pref medium
	local 2002:2:: dev lo proto none metric 0 pref medium
	local 2002:2::2 dev lo proto none metric 0 pref medium
	2002:2::/120 dev eth2 proto kernel metric 256 pref medium
	local fe80:: dev lo proto none metric 0 pref medium
	local fe80:: dev lo proto none metric 0 pref medium
	local fe80::ff:fe00:202 dev lo proto none metric 0 pref medium
	local fe80::ff:fe00:203 dev lo proto none metric 0 pref medium
	fe80::/64 dev eth1 proto kernel metric 256 pref medium
	fe80::/64 dev eth2 proto kernel metric 256 pref medium
	ff00::/8 dev vrf-red metric 256 pref medium
	ff00::/8 dev eth1 metric 256 pref medium
	ff00::/8 dev eth2 metric 256 pref medium


	8. Route Lookup for a VRF

	A test route lookup can be done for a VRF by adding the oif option to ip:
	$ ip [-6] route get oif VRF-NAME ADDRESS

	For example:
	$ ip route get 10.2.1.40 oif vrf-red
	10.2.1.40 dev eth1 table vrf-red src 10.2.1.2
	cache

	$ ip -6 route get 2002:1::32 oif vrf-red
	2002:1::32 from :: dev eth1 table vrf-red proto kernel src 2002:1::2 metric 256 pref medium


	9. Removing Network Interface from a VRF

	Network interfaces are removed from a VRF by breaking the enslavement to
	the VRF device:
	$ ip link set dev NAME nomaster

	Connected routes are moved back to the default table and local entries are
	moved to the local table.

	For example:
	$ ip link set dev eth0 nomaster

	--------------------------------------------------------------------------------

	Commands used in this example:

	cat >> /etc/iproute2/rt_tables <<EOF
	1 vrf-mgmt
	10 vrf-red
	66 vrf-blue
	81 vrf-green
	EOF

	function vrf_create
	{
	VRF=$1
	TBID=$2
	# create VRF device
	ip link add vrf-${VRF} type vrf table ${TBID}

	# add rules that direct lookups to vrf table
	ip ru add pref 200 oif vrf-${VRF} table ${TBID}
	ip ru add pref 200 iif vrf-${VRF} table ${TBID}
	ip -6 ru add pref 200 oif vrf-${VRF} table ${TBID}
	ip -6 ru add pref 200 iif vrf-${VRF} table ${TBID}

	if [ "${VRF}" != "mgmt" ]; then
	ip route add table ${TBID} prohibit default
	fi
	ip link set dev vrf-${VRF} up
	ip link set dev vrf-${VRF} state up
	}

	vrf_create mgmt 1
	ip link set dev eth0 master vrf-mgmt

	vrf_create red 10
	ip link set dev eth1 master vrf-red
	ip link set dev eth2 master vrf-red
	ip link set dev eth5 master vrf-red

	vrf_create blue 66
	ip link set dev eth3 master vrf-blue

	vrf_create green 81
	ip link set dev eth4 master vrf-green


	Interface addresses from /etc/network/interfaces:
	auto eth0
	iface eth0 inet static
	address 10.0.0.2
	netmask 255.255.255.0
	gateway 10.0.0.254

	iface eth0 inet6 static
	address 2000:1::2
	netmask 120

	auto eth1
	iface eth1 inet static
	address 10.2.1.2
	netmask 255.255.255.0

	iface eth1 inet6 static
	address 2002:1::2
	netmask 120

	auto eth2
	iface eth2 inet static
	address 10.2.2.2
	netmask 255.255.255.0

	iface eth2 inet6 static
	address 2002:2::2
	netmask 120

	auto eth3
	iface eth3 inet static
	address 10.2.3.2
	netmask 255.255.255.0

	iface eth3 inet6 static
	address 2002:3::2
	netmask 120

	auto eth4
	iface eth4 inet static
	address 10.2.4.2
	netmask 255.255.255.0

	iface eth4 inet6 static
	address 2002:4::2
	netmask 120