Instead of having a template published, the integration creates a template that matches the version of the integration that is being installed.
You will run the
cd ~
acc-provision -f kubernetes-1.15 --sample > aci-fabric-config.yaml
To assist with the process of creating this template file, we will breakdown the file components and explain them. Following that explanation we will provide a single command to create the file for you based on the data specific to your POD.
One of the most important key components to get the integration working is properly configuring the ACI PROVISION yaml file. This is divided into various sections.
The first section provides the system_id that is unique per Kubernetes cluster
#
# Configuration for ACI Fabric
#
aci_config:
system_id: [value]
apic_hosts:
- [IP ADDRESS OF APIC]
Field | Description |
---|---|
system_id | The system_id must be unqie per kubernetes cluster interacting with an ACI fabric. It is used to identify many policy configurations in the ACI fabric. |
apic_hosts | List of IP addresses of the APIC controllers. |
vmm_domain:
encap_type: vxlan/vlan
mcast_range:
start: X.X.X.X
end: X.X.X.X
nested_inside:
type: vmware
name: [value]
Field | Description |
---|---|
vmm_domain | Configuration parameters for the Kubernetes VMM domain |
encap_type | The encapsulation that will be used by OPFLEX in between the fabric leaf and the openvswitch container running on the host. Can be VXLAN or can be VLAN. |
mcast_range | For every VMM domain in an ACI fabric, you must specify a specific multicast range that it can use. This controls possible BUM flood traffic problems by keeping that into specific multi-cast topology trees inside the ACI fabric. |
nested_inside | If you are going to run kubernetes inside a virtualized platform like VMware or OpenStack, you have to define the specific type and the name of the VMM domain that you are going to utilize. |
aep: [value]
vrf:
name: [value]
tenant: [value]
l3out:
name: [value]
external_networks:
- [value]
Field | Description |
---|---|
aep | The name of the Attachable Entity Profile created in the ACI fabric for the domain ( either physical or virtual). The AEP will be used to connect the access policies to the application profile policies in the ACI fabric. |
vrf | You have to create the VRF in the ACI fabric and then provide the name for the VRF in the definition. This is defined in the common tenant. |
tenant | The tenant in which the application profiles will be built. |
l3out | The name of the L3OUT that will be used by the integration for external services. Name needs to match exactly and is defined in common. |
external networks | This is the defined L3OUT EPG for the provided L3 OUT. |
net_config:
#Internal Subnets
pod_subnet: [value]
node_svc_subnet: [value]
#External Visible Subnets
node_subnet: [value]
extern_dynamic: [value]
extern_static: [value]
# VLAN Identifiers
kubeapi_vlan: [value]
service_vlan: [value]
infra_vlan: [value]
Field | Description |
---|---|
pod_subnet | The POD subnet is probably the largest address space required by kubernetes. Each POD will be assigned an IP address from this space and this address is only internal to the kubernetes fabric ( no external visibility ). e.g. 10.1.0.1/16 |
node_svc_subnet | The NODE service subnet is utilized by the ACI fabric for service graphs. The size of this subnet is directly related to the amount of nodes created in the fabric ( each node will have a specific IP address assigned ) e.g. 10.2.0.1/16 |
node_subnet | The tenant in which the application profiles will be built. |
extern_dynamic | The extern dynamic definition tells the ACI/Kubernetes integration what subnet to utilize for assigning dynamic allocation of external IP addresses. This will probably be a larger subnet that DEVOPS engineers will utilize more ad-hoc for external service definition |
extern_static | The extern static subnet gives the ACI/Kubernetes integration the subnet to utilize for statically defined external IP address services. These are usually tied back into specific DNS names and can be used by ingress controllers for wildcard DNS services. |
kubeapi_vlan | This is the VLAN encapsulation that will be used by the kubernetes nodes to talk to eachother for kubernetes API calls. This VLAN and subnet will communicate over the ACI fabric and will allow internal kubernetes API communications to happen. |
service_vlan | The service VLAN which should not be confused with the SERVICE CIDR for kubernetes is utilized by the ACI fabric for all the service graph PBR functionality in the fabric. |
infra_vlan | This is the ACI fabric infrastructure VLAN. This is configured during fabric initialization and can be extracted from your fabric under the infra tenant. |
node_config:
uplink_iface: [value]
vxlan_uplink_iface: [value]
Field | Description |
---|---|
uplink_iface | The primary interface connecting to the ACI fabric |
vxlan_uplink_iface | The subinterface towards the infra VLAN towards the ACI fabric. |
registry:
image_prefix: noiro
# image_pull_secret: secret_name
Field | Description |
---|---|
image_prefix | If you have a requirement to install your own private container registry due to the kubernetes cluster not being able to reach external networks, you can download the containers from Cisco.com and create your private registry. This would be the prefix information. |
image_pull_secret | Password definition for private registry. |
To simplify the lab we have taken all the proper parameters for your specific LAB POD and configured the deployment for you directly.
cat << EOF > ~/aci-fabric-config.yaml
#
# Configuration for ACI Fabric
#
aci_config:
system_id: k8s_pod09 # Every opflex cluster must have a distict ID
apic_hosts: # List of APIC hosts to connect for APIC API
- 10.0.226.41
vmm_domain: # Kubernetes container domain configuration
encap_type: vxlan # Encap mode: vxlan or vlan
mcast_range: # Every opflex VMM must use a distinct range
start: 239.9.1.1
end: 239.9.255.255
nested_inside: # Include if nested inside a VMM
type: vmware # Specify the VMM vendor (supported: vmware)
name: k8s_pod09_dvs # Specify the name of the VMM domain
# The following resources must already exist on the APIC,
# they are used, but not created by the provisioning tool.
aep: k8s_pod09_aep # The AEP for ports/VPCs used by this cluster
vrf: # This VRF used to create all kubernetes EPs
name: k8s_vrf
tenant: common # This can be system-id or common
l3out:
name: k8s # Used to provision external IPs
external_networks:
- k8s-epg # Used for external contracts
#
# Networks used by Kubernetes
#
net_config:
#Internal Subnets
pod_subnet: 10.209.0.1/16 # Subnet to use for Kubernetes Pods
node_svc_subnet: 10.109.2.1/24 # Subnet to use for service graph
#External Visible Subnets
node_subnet: 10.0.144.65/29 # Subnet to use for nodes
extern_dynamic: 10.0.145.65/29 # Subnet to use for dynamic external IPs
extern_static: 10.0.146.65/29 # Subnet to use for static external IPs
# VLAN Identifiers
kubeapi_vlan: 3109 # The VLAN used by the physdom for nodes
service_vlan: 3209 # The VLAN used by LoadBalancer services
infra_vlan: 3967 # The VLAN used by ACI infra
#node_config:
# uplink_iface: ens224
# vxlan_uplink_iface: ens224.3967
#
# Configuration for container registry
# Update if a custom container registry has been setup
#
registry:
image_prefix: noiro # e.g: registry.example.com/noiro
# image_pull_secret: secret_name # (if needed)
EOF
cd ~
acc-provision -c aci-fabric-config.yaml -o aci-k8s-config.yaml -f kubernetes-1.15 -a -u admin -p cisco.123
The output will look similar to the following
INFO: Loading configuration from "aci-fabric-config.yaml" INFO: Using configuration flavor kubernetes-1.15 INFO: Generating certs for kubernetes controller INFO: Private key file: "user-k8s_pod17.key" INFO: Certificate file: "user-k8s_pod17.crt" INFO: Provisioning configuration in APIC WARN: User already exists (k8s_pod17), recreating user INFO: Using configuration label aci-containers-config-version=d7c16dce-f605-4cb9-b18a-46d35a8733b2 INFO: Writing kubernetes infrastructure YAML to aci-k8s-config.yaml INFO: Apply infrastructure YAML using: INFO: kubectl apply -f aci-k8s-config.yaml INFO: Delete stale objects from older deployments using: INFO: kubectl -n kube-system delete configmap,secret,serviceaccount,daemonset,deployment,clusterrolebinding,clusterrole -l 'aci-containers-config-version,aci-containers-config-version notin (d7c16dce-f605-4cb9-b18a-46d35a8733b2)'