Master Node Setup

Master
Setup linux!
1
Introduction
2
Understanding Containers
3
ACI Access Policies
4
VMware Domain Definition
5
Linux Host Setup
6
ACI Kubernetes
7
Configure VMware Integration
8
Initialize Kubernetes
9
Deploy Applications
10
Conclusion
11
Reference

With the completion of configuration of Access policies in the ACI fabric, you now have to setup the three separate Linux Virtual Machines we have created for you in this lab. To this point the only thing we have done is place the virtual machines on the ESXi compute host and attach the cables to the ACI fabric.

Some of these steps might seem redundant that you completed previously in the docker section. We include these more for reference on how you would have to implement in your premise equipment.

What comes next is to setup the three separate CentOS Linux machines in a way that they are ready for when the Kubernetes integration completes the configuration in ACI for you.

Step 1 - SELinux

You have to disable SELINUX for container networking to work. If you are installing your own box make sure to disable SELINUX. For this lab, we have disabled SELINUX in PXE boot process itself to save time.

Step 2 - Disable SWAP on Linux

SWAP must be disabled for kubernetes.


swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

Step 3 - Install yum repository tools


yum install -y yum-utils

Step 4 - Install extra packages


yum -y install epel-release 

Step 5 - Install Python Package management tools


yum -y install python-wheel python-pip

Step 6 - Disable Services

There are some security services that need to be removed from the system before installing kubernetes. These include SELinux in RHAT world, iptables and NetworkManager. For expediency we have installed these during the boostrap processs. We show these commands for RHAT/Centos so you have a reference:

    setenforce 0
    systemctl disable iptables-services firewalld
    systemctl stop iptables-services firewalld
    systemctl disable NetworkManager
    systemctl stop NetworkManager

Step 7 - Enable IP forwarding


echo "net.bridge.bridge-nf-call-ip6tables=1" >> /etc/sysctl.conf
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-iptables=1" >> /etc/sysctl.conf

Step 8 - Setup Uplink interface

Since we are doing the integration with VMware, you will have a single interface that is connected to the WMware DVS that will then connect to the fabric redundantly through ESXi.



cat << EOF > /etc/sysconfig/network-scripts/ifcfg-ens224
DEVICE=ens224
HWADDR=
IPV6INIT=no
NAME=ens224
ONBOOT=yes
MTU=9000
EOF

Uplink Interface

The uplink interface connecting the Kubernetes host to the ACI fabric will be communicating using OpFlex to the ACI fabric.

The sub-interface will use a VLAN encapsulated interface on the interface pointing to the ACI fabric. The VLAN that will be utilized, is called the infra VLAN in ACI. The infra VLAN is a construct of ACI used to extend the internal overlay-1 infrastructure of the ACI fabric to devices that are not attached directly to the spines. When configuring the ACI fabric initialization process, the specific value of the infra VLAN is defined by the fabric administrator.

We recommend never using VLANS that are infrastructure VLANs in Nexus7k, Nexus5k. These might lead to interface VLAN problems when integrating with brownfield standalone switches. Also to notice is that the configuration of the infra VLAN in ACI is done during fabric initialization and cannot be modified without initializing the whole fabric.

Once the L2 encapsulation layer is done between the ACI fabric port and the linux compute node, the bond interface can retrieve it's IP address in infra VLAN from the fabric via DHCP. This is the reason that we first configured access policies to interact with the APIC controller and provided the physical ports that this node is attached to the ACI fabric. ACI is a white list fabric model and in any such model we have to first enable the policy model in ACI so that when the linux compute host requests it's DHCP via the infrastructure VLAN, the network constructs are ready to respond.

Step 9 - Configure OpFlex interface


cat << EOF > /etc/sysconfig/network-scripts/ifcfg-ens224.3967
PERSISTENT_DHCLIENT=1
DHCPRELEASE=1
DEVICE=ens224.3967
ONBOOT=yes
PEERDNS=yes
NM_CONTROLLED=no
HWADDR=`ip link show ens224 | awk '/ether/ {print $2}'`
TYPE=Ethernet
BOOTPROTO=dhcp
VLAN=yes
ONPARENT=yes
MTU=1600
EOF

Step 10 - Add Routes


cat << EOF > /etc/sysconfig/network-scripts/route-ens224.3967
ADDRESS0=224.0.0.0
NETMASK0=240.0.0.0
INTERFACE=ens224.3967
METRIC0=0
EOF

Step 11 - Add Infra Sub-Interface DHCP Options


cat << EOF > /etc/dhcp/dhclient-ens224.3967.conf
send dhcp-client-identifier 01:`ip link show ens224 | awk '/ether/ {print $2}'`;
request subnet-mask, domain-name, domain-name-servers, host-name;
send host-name `hostname | cut -d "." -f 1`;

option rfc3442-classless-static-routes code 121 = array of unsigned integer 8;
option ms-classless-static-routes code 249 = array of unsigned integer 8;
option wpad code 252 = string;

also request rfc3442-classless-static-routes;
also request ms-classless-static-routes;
also request static-routes;
also request wpad;
also request ntp-servers;
EOF

Step 12 - Create sub-interfaces for k8s Node INFRA across the fabric

Of the defined two sub-interfaces, this sub-interface is what will be used for the kubernetes nodes to talk between themselves for control-plane traffic. These are API calls made from the kubernetes worker nodes to either a master node, etc node or others. Looking at the diagram we saw again, the sub-interface vlan is 3109.


All the Kubernetes nodes will need the assignment of a physical IP as there is no DHCP functionality. In your network you should consider leveraging the right subnet mask in order to be scale in your environment. For this lab we will just build a /29 subnet since each student will do a separate integration and have minimal quantity of compute nodes.


cat << EOF > /etc/sysconfig/network-scripts/ifcfg-ens224.3109
DEVICE=ens224.3109
IPADDR=10.0.144.67
PREFIX=29
ONBOOT=yes
PEERDNS=yes
NM_CONTROLLED=no
VLAN=yes
ONPARENT=yes
MTU=9000
EOF

Step 13 - Add static route for k8s node infra


cat << EOF > /etc/sysconfig/network-scripts/route-ens224.3109
ADDRESS0=10.209.0.0
NETMASK0=255.255.0.0
GATEWAY0=10.0.144.65
EOF

Step 14 - Add google reposotories

The repository needs to be added for CentOS distribution because it is NOT included by default. The following command will paste into the file the repository information.


cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

Since we didn't use yum-config-manager, we have to tell yum to update the cache after adding a remote destination.


yum -y makecache fast

Step 15 - Install Kubernetes


yum install -y kubernetes-cni-0:0.7.5-0.x86_64 kubelet-0:1.15.7-0.x86_64 kubectl-0:1.15.7-0.x86_64 kubeadm-0:1.15.7-0.x86_64

Step 16 - Enable and Start the kubelet service


systemctl enable kubelet.service


systemctl start kubelet.service