With the completion of configuration of Access policies in the ACI fabric, you now have to setup the three separate Linux Virtual Machines we have created for you in this lab. To this point the only thing we have done is place the virtual machines on the ESXi compute host and attach the cables to the ACI fabric.
What comes next is to setup the three separate CentOS Linux machines in a way that they are ready for when the Kubernetes integration completes the configuration in ACI for you.
You have to disable SELINUX for container networking to work. If you are installing your own box make sure to disable SELINUX. For this lab, we have disabled SELINUX in PXE boot process itself to save time.
SWAP must be disabled for kubernetes.
swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
yum install -y yum-utils
yum -y install epel-release
yum -y install python-wheel python-pip
There are some security services that need to be removed from the system before installing kubernetes. These include SELinux in RHAT world, iptables and NetworkManager. For expediency we have installed these during the boostrap processs. We show these commands for RHAT/Centos so you have a reference:
setenforce 0 systemctl disable iptables-services firewalld systemctl stop iptables-services firewalld systemctl disable NetworkManager systemctl stop NetworkManager
echo "net.bridge.bridge-nf-call-ip6tables=1" >> /etc/sysctl.conf
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-iptables=1" >> /etc/sysctl.conf
Since we are doing the integration with VMware, you will have a single interface that is connected to the WMware DVS that will then connect to the fabric redundantly through ESXi.
cat << EOF > /etc/sysconfig/network-scripts/ifcfg-ens224
DEVICE=ens224
HWADDR=
IPV6INIT=no
NAME=ens224
ONBOOT=yes
MTU=9000
EOF
The uplink interface connecting the Kubernetes host to the ACI fabric will be communicating using OpFlex to the ACI fabric.
The sub-interface will use a VLAN encapsulated interface on the interface pointing to the ACI fabric. The VLAN that will be utilized, is called the infra VLAN in ACI. The infra VLAN is a construct of ACI used to extend the internal overlay-1 infrastructure of the ACI fabric to devices that are not attached directly to the spines. When configuring the ACI fabric initialization process, the specific value of the infra VLAN is defined by the fabric administrator.
We recommend never using VLANS that are infrastructure VLANs in Nexus7k, Nexus5k. These might lead to interface VLAN problems when integrating with brownfield standalone switches. Also to notice is that the configuration of the infra VLAN in ACI is done during fabric initialization and cannot be modified without initializing the whole fabric.
Once the L2 encapsulation layer is done between the ACI fabric port and the linux compute node, the bond interface can retrieve it's IP address in infra VLAN from the fabric via DHCP. This is the reason that we first configured access policies to interact with the APIC controller and provided the physical ports that this node is attached to the ACI fabric. ACI is a white list fabric model and in any such model we have to first enable the policy model in ACI so that when the linux compute host requests it's DHCP via the infrastructure VLAN, the network constructs are ready to respond.
cat << EOF > /etc/sysconfig/network-scripts/ifcfg-ens224.3967
PERSISTENT_DHCLIENT=1
DHCPRELEASE=1
DEVICE=ens224.3967
ONBOOT=yes
PEERDNS=yes
NM_CONTROLLED=no
HWADDR=`ip link show ens224 | awk '/ether/ {print $2}'`
TYPE=Ethernet
BOOTPROTO=dhcp
VLAN=yes
ONPARENT=yes
MTU=1600
EOF
cat << EOF > /etc/sysconfig/network-scripts/route-ens224.3967
ADDRESS0=224.0.0.0
NETMASK0=240.0.0.0
INTERFACE=ens224.3967
METRIC0=0
EOF
cat << EOF > /etc/dhcp/dhclient-ens224.3967.conf
send dhcp-client-identifier 01:`ip link show ens224 | awk '/ether/ {print $2}'`;
request subnet-mask, domain-name, domain-name-servers, host-name;
send host-name `hostname | cut -d "." -f 1`;
option rfc3442-classless-static-routes code 121 = array of unsigned integer 8;
option ms-classless-static-routes code 249 = array of unsigned integer 8;
option wpad code 252 = string;
also request rfc3442-classless-static-routes;
also request ms-classless-static-routes;
also request static-routes;
also request wpad;
also request ntp-servers;
EOF
Of the defined two sub-interfaces, this sub-interface is what will be used for the kubernetes nodes to talk between themselves for control-plane traffic. These are API calls made from the kubernetes worker nodes to either a master node, etc node or others. Looking at the diagram we saw again, the sub-interface vlan is 3109.
All the Kubernetes nodes will need the assignment of a physical IP as there is no DHCP functionality. In your network you should consider leveraging the right subnet mask in order to be scale in your environment. For this lab we will just build a /29 subnet since each student will do a separate integration and have minimal quantity of compute nodes.
cat << EOF > /etc/sysconfig/network-scripts/ifcfg-ens224.3109
DEVICE=ens224.3109
IPADDR=10.0.144.67
PREFIX=29
ONBOOT=yes
PEERDNS=yes
NM_CONTROLLED=no
VLAN=yes
ONPARENT=yes
MTU=9000
EOF
cat << EOF > /etc/sysconfig/network-scripts/route-ens224.3109
ADDRESS0=10.209.0.0
NETMASK0=255.255.0.0
GATEWAY0=10.0.144.65
EOF
The repository needs to be added for CentOS distribution because it is NOT included by default. The following command will paste into the file the repository information.
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
Since we didn't use yum-config-manager
, we have to tell yum to update the cache after
adding a remote destination.
yum -y makecache fast
yum install -y kubernetes-cni-0:0.7.5-0.x86_64 kubelet-0:1.15.7-0.x86_64 kubectl-0:1.15.7-0.x86_64 kubeadm-0:1.15.7-0.x86_64
systemctl enable kubelet.service
systemctl start kubelet.service