Understanding ACI PBR

Mapping Services to PODS

When you created the service, the integration between ACI and Kubernetes made the ACI constructs and policies to expose the external IP. To accomplish this goal ACI utilized PBR (policy based redirect). PBR makes it possible for the ACI fabric to redirect traffic into L4/L7 devices without the need for these devices to be the default gateway or routep point. This is popular construct in networks for firewalls, intrussion prevention systems, load balancers and more.

Since PBR is loaded on top of ACI contracts you can also do policy enforcement on the services that are being configured. For example, in the configuration you can create network filters so that traffic destined to a web service are only allowed to hit port 80 and 443. Making sure that the load balancer is receiving only traffic that policy allows for a more secure network.

The below diagram is a representation of what the integration between ACI and Kubernetes built to provide the service you defined to be exposed for the simple MyLabAPP. This provides you with an easy ability to load balance in your enterprise private network services that are developed for the platform. Since ACI will do the load balancing, as you scale the containers in Kubernetes to handle the service load capacity, ACI automatically will redirect the traffic into the right containers.

Since there is no defined bridge domain that would contain the subnet that is defined for the dynamic service, you have to build a static route from the adjacent router into the ACI fabric. For this lab we just pre-built the static routes for you to simplify the lab complexity.

The configuration that is deployed in ACI starts in the External Layer 3 policy object. Looking in the ACI fabric under the common tenant you can find this policy.

The service assigned IP should be the same as what you have in value when doing the command kubectl get pods -o wide. Another easy way to get this information is to access this via the ACI fabric APIC.

The ACI APIC leverages the same NX-OS CLI that can be used to view the fabric objects just like you can view these from the web interface. In the Credentials link on the top of this web page you have a link to connect via SSH to the APIC.

show tenant common vrf k8s_vrf external-l3 epg k8s_pod09_svc_default_mylabapp detail

 Name            Flags                      Match               Node        Entry               Oper State 
 --------------  -------------------------  ------------------  ----------  ------------------  ---------- 
 common:         vxlan: 2555909         node-209     enabled    
 k8s_pod09_svc_  vrf: k8s_vrf                                   node-207     enabled    
 default_mylaba  Target dscp: unspecified                       node-210     enabled    
 pp              qosclass: unspecified                          node-208     enabled    
                 Provided: k8s_pod09_svc_d                                                                 

Using the same CLI we can also look at the created graph in the fabric.

show l4l7-graph tenant common graph k8s_pod09_svc_global

Graph           : common-k8s_pod09_svc_global
Graph Instances : 1                          

Consumer EPg  : common-k8s-epg                       
Provider EPg  : common-k8s_pod09_svc_default_mylabapp
Contract Name : common-k8s_pod09_svc_default_mylabapp
Config status : applied                              

Function Node Name : loadbalancer
Service Redirect   : enabled
 Connector   Encap       Bridge-Domain  Device Interface      Service Redirect Policy        
 ----------  ----------  ----------     --------------------  ------------------------------ 
 provider    vlan-3209   common-k8s_po  interface             k8s_pod09_svc_default_mylabapp 
 consumer    vlan-3209   common-k8s_po  interface             k8s_pod09_svc_default_mylabapp 

As you can see the ACI/Kubernetes integration has created a bridge domain that is used for the definition of the service graph instances in the fabric.

Looking at the diagram we showed earlier, we see that the contract that is created between the EPG for the service and the external network is where the Service Graph is layered upon.

Looking in the APIC CLI we can see:

show tenant common contract k8s_pod09_svc_default_mylabapp

 Tenant      Contract    Type        Qos Class     Scope       Subject     Access-group  Dir   Description 
 ----------  ----------  ----------  ------------  ----------  ----------  ----------    ----  ----------  
 common      k8s_pod09_  permit      unspecified   vrf         loadbalanc  k8s_pod09_sv  both              
             svc_defaul                                        edservice   c_default_my                    
             t_mylabapp                                                    labapp           

One of the important value statements of the ACI integration with Kubernetes, is how service definitions in Kubernetes translates directly into ACI policies to be implemented in the fabric dynamically. If you look at the following example, any of the specified ports that are defined in the Kubernetes service YAML file will be created in the ACI fabric.

apiVersion: v1
kind: Service
  name: mycrazyapp
    app: mycrazyapp
  type: LoadBalancer
  - name: http
    port: 80
    targetPort: 80
  - name: https
    port: 443
    targetPort: 443
  - name: mgmt
    port: 8080
    targetPort: 8080
  - name: data-ingress
    port: 9183
    targetPort: 9183
    app: mycrazyapp

This provides us the detail that the access-group/filter is called k8s_pod09_svc_default_mylabapp and we could verify the filter that we are using for this particular application which in this case is TCP port 80:

show tenant common access-list k8s_pod09_svc_default_mylabapp  
apic1# show tenant common access-list k8s_pod09_svc_default_mylabapp
Tenant : common
Access-List : k8s_pod09_svc_default_mylabapp

  match tcp dest 80

You can also look at at the PBR services created in ACI based on the Kubernetes deployment. You will be able to see the hashing algorithm which in this case uses the default setting sip-dip-prototype. This means this policy will be hashing the traffic based on Source IP + Destination IP + Protocol Type.

This window also shows the destination nodes which are the services nodes. In your case it would be the IP address assigned to pod09-node1 and pod09-node2 from the OpFlex protocol.

Back in the ACI fabric:

Packet Flow

As part of the configuration for the integration to work you have to define the static route that points towards the ACI fabric leaf that has the defined L3 out. Traffic that arrives from the network towards the exposed service IP will be routed via this mechanism. The ACI fabric will never advertise the route to the network by itself.

When the packet hits the ACI fabric leaf, the L3 out policy object will probably have a default route defined leaving the fabric back towards the adjacent router. If it doesn't, this won't change the behaviour we are going to explain.

As the packet is in the L3 out object context, it would look in the VRF for a destination to send the packet and this will trigger a MISS since it is not defined anywhere in a bridge domain. For this reason, before sending the packet back out it performs a look up on the SRC IP and classifies the packet into the default EPG that has defined as destination.

Now the packet should be ready to be sent out the fabric, but when ACI does the final lookup to send the packet; Longest Prefix Match (LPM) places the packet in the service EPG due to match of the DST IP to the /32 in the service EPG.

Since LPM placed this in the external service EPG that ACI/Kubernetes built the policy forces the traffic into the Service Graph and leverages the Contract that was created (in our case TCP 80) and sends it to the the Policy Based Redirect (PBR) construct. At this point, the service graph knows which destination nodes this packet is for as defined in the policy.

Once the packet reaches the destination node, the packet is then Network Address Translated (NAT) by openvswitch (OVS) that is running on that node. That operation converts the SERVICE IP destination IP address to the POD Destination IP and routed in the host. If there are various PODS on the same compute host for the servive, OVS can also do a final load balancing locally towards the various POD IP's.

In a large fabric you can observe the value of ACI managing this process for you. Any increment in the count of scalability would create more pods to manage that service. Each would get address to the service graph and the PBR would properly load balance across the fabric.

One last part of the life of the packet is that the container is listening in a service that is different than what is being sent to it. If you recall in the previous example the container was listening on port 8090. So for this solution to work, NAT also has to perform Protocol address translation. In this case when you defined the service the target port was specified and this is all the information that is needed for the ACI/Kubernetes integration to program the OpenVSwitch to perform this task.