MyLabAPP in k8s


In the docker section we showed you a very simple application that was a container running Flask and presenting you the IP of the container. Now we are going to deploy the same application inside of Kubernetes

Step 1 - Create MyApp deployment file

In kubernetes you can create deployments in the CLI directly or you can define the deployments in YAML files. Using YAML files to configure these different kubernetes components is what is called declarative configuration. The advtange of using this method is that it drives directly into two important parts of kubernetes.

  • Consistency in how applications are deployed
  • Automation of deployment of applications
For this lab we will be using YAML files to easily create and delete resources as needed in the cluster.

cat << EOF > ~/mylabapp-deployment.yaml
apiVersion: apps/v1
kind: Deployment
  name: mylabapp
    app: mylabapp
      app: mylabapp
    type: Recreate
        app: mylabapp
      - image:
        name: mylabapp
        - containerPort: 8090
          name: mylabapp

Step 2 - Deploy mylabapp

With the YAML definition completed, we can create the deployment.

kubectl create -f ~/mylabapp-deployment.yaml 

The will create the kubernetes deployment.

deployment.apps/mylabapp created

You can describe the deployment to get more information.

kubectl describe deployment mylabapp

The output of the command gives us details on the deployment that was built.

Name:               mylabapp
Namespace:          default
CreationTimestamp:  Tue, 22 May 2018 09:22:12 -0400
Labels:             app=mylabapp
Selector:           app=mylabapp
Replicas:           1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:       Recreate
MinReadySeconds:    0
Pod Template:
  Labels:  app=mylabapp
    Port:         8090/TCP
    Environment:  [none]
    Mounts:       [none]
  Volumes:        [none]
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  [none]
NewReplicaSet:   mylabapp-767db98d99 (1/1 replicas created)
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  58s   deployment-controller  Scaled up replica set mylabapp-767db98d99 to 1

Integration of Kubernetes POD object to the ACI fabric.

Using the describe command from kubectl you can get a glimpse at important information that has been configured in the pod.

kubectl describe pods mylabapp

Name:           mylabapp-767db98d99-94bvn
Namespace:      default
Start Time:     Tue, 22 May 2018 09:22:15 -0400
Labels:         app=mylabapp
Status:         Running
IP:             10.X.X.X
Controlled By:  ReplicaSet/mylabapp-767db98d99
    Container ID:   docker://5708ccdd9e30b2adbfc95cb744969ebcbded5c27faf44583a194a36162a1c7d8
    Image ID:       docker-pullable://
    Port:           8090/TCP
    State:          Running
      Started:      Tue, 22 May 2018 09:22:32 -0400
    Ready:          True
    Restart Count:  0
    Environment:    [none]
      /var/run/secrets/ from default-token-td2gp (ro)
  Type           Status
  Initialized    True
  Ready          True
  PodScheduled   True
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-td2gp
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  [none]
Tolerations: for 300s
        for 300s
  Type    Reason                 Age   From                                        Message
  ----    ------                 ----  ----                                        -------
  Normal  Scheduled              15m   default-scheduler                           Successfully assigned mylabapp-767db98d99-94bvn to
  Normal  SuccessfulMountVolume  15m   kubelet,  MountVolume.SetUp succeeded for volume "default-token-td2gp"
  Normal  Pulling                15m   kubelet,  pulling image ""
  Normal  Pulled                 15m   kubelet,  Successfully pulled image ""
  Normal  Created                15m   kubelet,  Created container
  Normal  Started                15m   kubelet,  Started container

In the output of the description for the POD you can see the annotations field. Here we can see now that the integration with ACI has placed this POD inside a specific policy-space in the fabric. How DOES that translate into the ACI fabric?

If you head over the ACI Fabric you will see a Tenant for this LAB POD named k8s_pod09. Inside the application profile are now a series of End Point Groups (EPGs), including kube-default.

As you can see now the ACI fabric is visibly aware of the created Kubernetes POD for the mylabapp. It also knows the IP address that was assigned to the Kubernetes POD and the host that the container is actively running on. This visibility lifts a veil of darkness from the kubernetes cluster towards the network administrator.

ACI has also created polcies in the ACI that make it possible for the different PODS to communicate in the fabric across the context of the kubernetes overlay.

The ACI/Kubernetes integration policy are implmented both in ACI Fabric and also in the host via OpenFlow controls. The ACI integration utilized the opflex protocol to communicate between the ACI fabric leaf and the compute host. This communication makes it possible to build the proper policies that extend these beyond just iptables in a single host.

ACI also creates a Virtual networking integration for the kubernetes cluster. In this we can see constructs of Kubernetes, like the deployment that was just created. These component visibility from the ACI fabric provides the network administrator granular visibility of what the DEVOPS teams are building.

Step 3 - Create quick POD to test.

You will now create a quick POD to test the reachability in the fabric. We will run it with the parameter --rm to delete the deployment when we are done and the parameter --it to interact with it. As we are dealing with a fabric overlay orchestrator, while the container is created in any particular host, you will be interacting with the container over the network.

kubectl run alpine --rm -it /bin/sh

Once inside the POD you will use the command: wget --spider --timeout=1 plus the IP address that you got from kubectl or ACI GUI that was assigned to the pod mylabapp. You will also have to specify the PORT as the application is listening on port 8090.

If you don't see a command prompt, try pressing enter.
/ # wget --spider --timeout=1 10.209.X.X:8090
Connecting to 10.209.X.X:8090 (10.209.X.X:8090)
/ # exit

This pod that we just created to run busybox connected from one pod to another pod in the kubernetes cluster over the ACI fabric.

Step 4 - Create service definition file

With the deployment enabled in the kubernetes cluster, we can create the service that we wish to expose to the outside world. With this service YAML definition you specify to kubernetes what the service is. In this case a LoadBalancer service.

In the file we have also defined a specific IP address that will be used. This is a slight variation that is done to assist in the lab manual to reach the service URL. In the real world the most prevalent option would be to let kubernetes assign a IP address from the pool. In some cases these are pre-assigned DNS names that make it easy for developers to reach endpoints.

cat << EOF > ~/mylabapp-service.yaml
apiVersion: v1
kind: Service
  name: mylabapp
    app: mylabapp
  type: LoadBalancer
  - port: 80
    targetPort: 8090
    app: mylabapp

Step 5 - Create service

kubectl create -f ~/mylabapp-service.yaml

Once completed you can list the services and pods that have been deployed.

kubectl get svc,pods -o wide

The output should be similar to the following with both a CLUSTER-IP and also a EXTERNAL-IP. For this student pod you should have the external IP address defined of

NAME             TYPE           CLUSTER-IP     EXTERNAL-IP        PORT(S)        AGE       SELECTOR
svc/mylabapp     LoadBalancer   10.X.X.X       80:XXXX/TCP    2m        app=mylabapp
svc/kubernetes   ClusterIP      [none]             443/TCP        2d        [none]

NAME                           READY     STATUS    RESTARTS   AGE       IP           NODE
po/mylabapp-6ddb9bfd5f-vv5l4   1/1       Running   0          3m        10.X.X.X

You can point your browser to that IP address to reach the deployed service.

Step 6 - Managing scale of application

One of the big advantages of the integration is that the scalability of the application is managed by kubernetes. You can easily ask kubernetes to increase the amount of applications and the load balancer service will properly adjust all policies to handle this.

kubectl scale --replicas=4 deployment mylabapp

Once kubernetes completes, you can look to see the deployments.

kubectl get svc,pods -o wide

NAME             TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)        AGE       SELECTOR
svc/mylabapp     LoadBalancer   10.96.XX.XX   80:30911/TCP   7m        app=mylabapp
svc/kubernetes   ClusterIP      [none]         443/TCP        6d        [none]

NAME                           READY     STATUS    RESTARTS   AGE       IP             NODE
po/mylabapp-587458cc8f-78t4x   1/1       Running   0          5s        10.209.X.X
po/mylabapp-587458cc8f-g8hxg   1/1       Running   0          5s        10.209.X.X
po/mylabapp-587458cc8f-hd8qg   1/1       Running   0          31m       10.209.X.X
po/mylabapp-587458cc8f-nqjfw   1/1       Running   0          5s        10.209.X.X

As we will show you in the next section, the load balancing of the service to these four kubernetes pods is handled by the ACI fabric itself using application service graphs.


Now that you have deployed a service and a kubernetes POD, let's take a little closer look on how ACI using OpFlex stores data on the compute components. OpFlex communication protocol is established between the ACI fabric leaf and the compute node. This protocol exchanges information from the ACI fabric to provide policy enforcement inside the compute node via OpenVSwitch. With this protocol the EPG, contracts and other ACI constructs are converted into kubernetes network policies on the host itself. This technology is the same used in the Cisco OpenStack integration with ACI.

The first step we are going to need is to get the UID of the service itself. Using the output modified -o yaml we will be getting the object representation of the service in kubernetes.

kubectl get service mylabapp -o yaml

apiVersion: v1
kind: Service
  creationTimestamp: 2018-06-10T21:14:08Z
    app: mylabapp
  name: mylabapp
  namespace: default
  resourceVersion: "1413"
  selfLink: /api/v1/namespaces/default/services/mylabapp
  uid: 36f34183-6cf3-11e8-bab8-005056182503
  clusterIP: 10.96.XX.XX
  externalTrafficPolicy: Cluster
  - nodePort: 30845
    port: 80
    protocol: TCP
    targetPort: 8090
    app: mylabapp
  sessionAffinity: None
  type: LoadBalancer
    - ip:

That UID defined is what you need to get the service opflex definition file. You need to copy that UID that is unique to your pod and then goto:

cd /var/lib/opflex-agent-ovs/services/
ls -la

-rw-r--r-- 1 root root 690 Jun 10 17:15 36f34183-6cf3-11e8-bab8-005056182503-external.service
-rw-r--r-- 1 root root 604 Jun 10 17:15 36f34183-6cf3-11e8-bab8-005056182503.service
-rw-r--r-- 1 root root 574 Jun 10 17:04 c7d33c70-6cf1-11e8-bab8-005056182503.service
-rw-r--r-- 1 root root 839 Jun 10 17:05 cbbb9c04-6cf1-11e8-bab8-005056182503.service

You should see a list of files and the one that matches the UID you saw in your output should be the one that provides the service information. The output looks like the following.

cat the uuid.service file that is specific to your output above
cat 36f34183-6cf3-11e8-bab8-005056182503.service
  "uuid": "36f34183-6cf3-11e8-bab8-005056182503",
  "domain-policy-space": "common",
  "domain-name": "k8s_vrf",
  "service-mode": "loadbalancer",
  "service-mapping": [
      "service-ip": "10.96.X.X",
      "service-proto": "tcp",
      "service-port": 80,
      "next-hop-ips": [
      "next-hop-port": 8090,
      "conntrack-enabled": true
  "attributes": {
    "app": "mylabapp",
    "name": "mylabapp",
    "namespace": "default",
    "service-name": "default_mylabapp"

The UID information is also available in the Cisco ACI fabric under the VMM domain for the service definition.

There is also the endpoint file definition that would be located in the directory /var/lib/opflex-agent-ovs/endpoints/. To make the connection you can get the pods with the command:

kubectl get pods -o wide | grep mylabapp

mylabapp-7bbc7bf597-58kft   1/1     Running   0          14m   none     none
mylabapp-7bbc7bf597-bnjz9   1/1     Running   0          14m   none     none
mylabapp-7bbc7bf597-vhlj9   1/1     Running   0          26m   None     none
mylabapp-7bbc7bf597-wxf5v   1/1     Running   0          14m   none     none

Of the list of pods pick one of them and describe the kubernetes pod. Make sure you note the worker node (node1 or node2) that this pod resides on. In the example below, the pod resides on node1 from the output above.

kubectl get pod [pod_id that is one of that looks like mylabapp-7bbc7bf597-58kft -o yaml
apiVersion: v1
kind: Pod
  creationTimestamp: "2020-01-27T09:34:02Z"
  generateName: mylabapp-7bbc7bf597-
    app: mylabapp
    pod-template-hash: 7bbc7bf597
  name: mylabapp-7bbc7bf597-58kft
  namespace: default
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: mylabapp-7bbc7bf597
    uid: 85e18de2-efe0-4912-8ec2-0846e7ea5dab
  resourceVersion: "128970"
  selfLink: /api/v1/namespaces/default/pods/mylabapp-7bbc7bf597-58kft
  uid: dea12256-f8c1-4c56-a209-4de21d69cb0d

For this next step, please go to the worker node (node1 or node2) that came from your output above. The copy/paste function below can be used on either node, but you must be on the node the pod you described is on.

cd /var/lib/opflex-agent-ovs/endpoints/

Find the file that matches that UUID. It will include additional information but it starts with the UUID. The contents of the file will be similar to the below if you cat the file's contents:

Replace the UUID with yours in the below output:

cat dea12256-f8c1-4c56-a209-4de21d69cb0d_31b5fc8269107b29637ab27c81f048ed1e0e8349467c194d5393f9c17d70fded_vetha81f5f4f.ep 
  "uuid": "dea12256-f8c1-4c56-a209-4de21d69cb0d_31b5fc8269107b29637ab27c81f048ed1e0e8349467c194d5393f9c17d70fded_vetha81f5f4f",
  "eg-policy-space": "k8s_pod17",
  "endpoint-group-name": "kubernetes|kube-default",
  "ip": [
  "mac": "26:eb:36:3a:67:5b",
  "access-interface": "vetha81f5f4f",
  "access-uplink-interface": "pa-vetha81f5f4f",
  "interface-name": "pi-vetha81f5f4f",
  "attributes": {
    "app": "mylabapp",
    "interface-name": "vetha81f5f4f",
    "namespace": "default",
    "pod-template-hash": "7bbc7bf597",
    "vm-name": "mylabapp-7bbc7bf597-58kft"

We have included these so you have another tool in the troubleshooting belt. These ar OpFlex related files and provide data to OpenVswitch integration and how OpenVSwitch builds the tables in the compute node based on these policies.