OpenShift Networking with Antrea

Antrea is an OpenSource CNI Plugin for Kubernetes. It has some good performance and features advantages compared to other CNI Plugins in the market.
Antrea leverage OVS as a DataPlane, which is widely adopted and high-performance programmable virtual switch.
From features prospective, Antrea extends native K8s NetworkPolicy to have more granular policy by adding hierarchy of security policies called “Tiers”, cluster wide policies, priorities, and more.
Other than the policy advantages, Antrea has great flexibility when it comes to the deployment options. It support inter-nodes encryption using IPSec, overlays, or no-encap modes.
Antrea is infrastructure agnostic, and it could run on top of any cloud, and it supports multiple Operating Systems including Windows. Thanks to the portability of Open vSwitch, Antrea can use the same data plane implementation on both Linux and Windows Kubernetes Nodes.
For more information about Antrea please visit https://antrea.io
If you don’t have any experience on Antrea, then please take a look to its architecture on antrea.io
Antrea just joined CNCF recently
https://www.cncf.io/sandbox-projects/
https://blogs.vmware.com/opensource/2021/05/05/antrea-joins-cncf-sandbox/
https://github.com/vmware-tanzu/antrea

In this post I will show how to easily deploy an OpenShift Cluster with Antrea using Antrea Operator.
I am using OpenShift IPI on vSphere, but same steps applies on any other Cloud Provider.

Please note that Antrea is already certified CNI for OpenShift with a certified Operator.
https://catalog.redhat.com/software/operators/detail/5fb647d793431a8f675c4e15
https://access.redhat.com/articles/5436171

Infrastructure Prerequisites

For this procedure, we need a DHCP Server and two DNS entries.

DNS Entries	DNS Entries Example	IP Address	IP Address Example
api.{clusterID}.{domain_name}	api.ocp.vmwdxb.com	API VIP	192.168.25.201
*.apps.{clusterID}.{domain_name}	*.apps.ocp.vmwdxb.com	Ingress VIP	192.168.25.202

Creating OpenShift Manifests

First we will create an OpenShift Installation Directory and create install-config.yaml

mkdir antrea-test
cd antrea-test
nano install-config.yaml

In the install-config.yaml, we need to add networking section with antrea as networkType.

apiVersion: v1
baseDomain: vmwdxb.com
compute:
- hyperthreading: Enabled
  name: worker
  replicas: 3
  platform:
    vsphere:
      cpus: 4
      coresPerSocket: 2
      memoryMB: 16384
      osDisk:
        diskSizeGB: 120
controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3
  platform:
    vsphere:
      cpus: 4
      coresPerSocket: 2
      memoryMB: 16384
      osDisk:
        diskSizeGB: 120
metadata:
  name: ocp
networking:
  networkType: antrea
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineCIDR: 192.168.25.0/24
  serviceNetwork:
  - 172.30.0.0/16
platform:
  vsphere:
    vcenter: vcsa-tkg-01.vmwdxb.com
    username: administrator@vsphere.local
    password: VCENTER-PWD
    datacenter: Datacenter
    defaultDatastore: OCP-VSAN
    fips: false
    network: OCP-Nodes-Segment
    cluster: Ali-OCP-Cluster
    apiVIP: 192.168.25.201
    ingressVIP: 192.168.25.202
pullSecret: '...'
sshKey: '...'

Please modify the inputs as per your environment and add the pullSecret and sshKey as per the standard OpenShift deployment.
Now we can create OpenShift Manifests, but before doing that, we should backup our install-config.yaml somewhere because it will be consumed and deleted in the manifests creation process.

cp install-config.yaml ../install-config.yaml.backup
openshift-install create manifests

Adding Antrea Operator Manifests

Now we will get the Antrea Operator Manifests from github

cd..
git clone https://github.com/vmware/antrea-operator-for-kubernetes.git

Before copying Antrea Manifests to OpenShift folder, we may need to do some changes. first we need to point to an Antrea Operator Image in the operator.yaml.
I published an image on aalidrees/antrea-operator. Feel free to use it.

nano antrea-operator-for-kubernetes/deploy/operator.yaml
- name: antrea-operator
# Replace this with the built image name
#  image: REPLACE_IMAGE
  image: aalidrees/antrea-operator:v0.0.9-test

Optionally Antrea config and version could be modified in below yaml.

nano antrea-operator-for-kubernetes/deploy/operator.antrea.vmware.com_v1_antreainstall_cr.yaml

Now we can copy Antrea Deploy folder to OpenShift Manifests Folder

cp antrea-operator-for-kubernetes/deploy/* antrea-test/manifests

Creating OpenShift Cluster

Now we can create our OpenShift Cluster with Antrea

cd antrea-test
openshift-install create cluster

Note: if the cluster is being deployed on vSphere, then we need to add vCenter Certificates before creating the cluster. You can find how to do that and other good steps related to creating OpenShift on vSphere.
https://veducate.co.uk/deploy-vsphere-openshift-machine-resources/

Ubuntu Example,

curl -O https://{vCenter_FQDN}/certs/download.zip
unzip download.zip
cp certs/win/* /usr/local/share/ca-certificates
update-ca-certificates

OCP 4.6 Installation Issues

Note: Antrea needs podCIDR in the nodes specifications to work, there is an issue in OCP4.6 that the podCIDR is not configured in the nodes specs. You may need to configure that manually during the bootstrap. This Problem is solved in Antrea 1.0 by delegating the Pods IPAM to Antrea itself.
example,
oc edit node ocp-xxxx-master-0
specs:
podCIDR: 10.128.0.0/23

oc edit node ocp-xxxx-master-1
specs:
podCIDR: 10.128.2.0/23

….
repeat for all nodes


OCP 4.7 Installation Issues

Can not access K8s API once it is migrated from Bootstrap VM to Master Nodes. This Problem is solved in Antrea 1.0 by delegating the Pods IPAM to Antrea itself.

Commands Summary

mkdir antrea-test 
cd antrea-test 
nano install-config.yaml
cp install-config.yaml ../install-config.yaml.backup
openshift-install create manifests
cd..
git clone https://github.com/vmware/antrea-operator-for-kubernetes.git
nano antrea-operator-for-kubernetes/deploy/operator.yaml
nano antrea-operator-for-kubernetes/deploy/operator.antrea.vmware.com_v1_antreainstall_cr.yaml
cp antrea-operator-for-kubernetes/deploy/* antrea-test/manifests
cd antrea-test
openshift-install create cluster

In this post I showed how to deploy an OpenShift 4.x cluster with Antrea as a CNI Plugin.
Thank you for Reading!