In this blog I will go through NSX-T 3.0 and OpenShift 4.4 integration using the new NSX-T Operator for OpenShift. The Blog is based on personal testing for the NSX-T Openshift Operator published here and it is not validated by both vendors.

The test was done on vSphere 7 with VSAN storage.

We can start deploying OpenShift using the deployment guide on vSphere.
https://docs.openshift.com/container-platform/4.4/installing/installing_vsphere/installing-vsphere.html

NSX-T will provide all the Networking and Security functionality needed for OpenShift such as connectivity, IPAM, NetworkPolicy, Load Balancing, OpenShift Routes/Ingress, and traffic visibility.

This blog assumes good level of understating for both OpenShift and NSX-T. The steps will go as below,

1. Configure DNS and Load Balancing
2. Prepare NSX-T Components
3. Configure NSX Container Plugin (NCP) and publish NCP and Operator Images
4. Install OpenShift with NSX-T Operator

Configure DNS and Load Balancing

The first step is a standard for OpenShift Deployments. Nothing changes for DNS requirements.
for Load Balancing, NSX-T Load Balancer could be used. We will need to create two Active Monitors, two Server Pools, and two Load Balancing VIPs with the same IP address. One for TCP-6443, and another one for TCP-22623. We can use NSX-T Dynamic Group Membership to configure the Server Pools as shown in below example.

Both api and api-int DNS entries should point to the VIPs IP address.

Prepare NSX-T Components

There are multiple components we need to create

1. T0 Gateway for North-South connectivity with Static Routes or BGP
2. T1 Gateway for OpenShift Nodes connectivity
3. L2 Segment for OpenShift Nodes connectivity
4. DHCP Server for OpenShift Bootstrap process

Depending on NCP configuration. The OpenShift Pods may be connectected to a separate T1-GW than the OpenShift Nodes.

Publish NCP and Operator Images and configure NCP

There are two images that we will need.

1. NCP Image
2. NSX-T OpenShift Operator Image

The NCP Image could be downloaded from https://my.vmware.com/web/vmware/downloads (needs my-vmware account). While the Operator one could be built as described here https://github.com/vmware/nsx-container-plugin-operator

Once both images are available, they will need to be published in some container registry as we will need to pint to them later.

Now lets clone the Operator repo to configure NCP and the Operator

$ git clone https://github.com/vmware/nsx-container-plugin-operator.git

We will need to edit two files under the deploy folder.

1. Edit deploy/operator.yaml to point to NCP and Operator images

26 image: OPERATOR_IMAGE
37 value: "NCP_IMAGE"

• Edit deploy/configmap.yaml with your environment variables. Below are the minimum that needs modification

117 nsx_api_managers = NSXT-MANAGER-IP
119 nsx_api_user = NSXT-USER
121 nsx_api_password = NSXT-PASSWORD
256 external_ip_pools = EXTERNAL_IP_CIDR
263 tier0_gateway = T0-GW-NAME
281 overlay_tz = OVERLAY-TRANSPORT-ZONE-UUID
348 edge_cluster = EDGE-CLUSTER-UUID
473 apiserver_host_ip = api-int.OPENSHIFT-CLUSTER.BASE-DOMAIN

Install OpenShift

There are couple of deviation from the standard OpenShift UPI deployment.

1. Modify install-config.yaml
2. Copy the operator “deploy” folder to OpenShift Manifests

Modify install-config.yaml

When deploying OpenShift, the first step is to create the install-config.yaml. we need to add “networking” section highlighted so OpenShift use NCP as a CNI instead of using openshift-sdn.

apiVersion: v1
baseDomain: vmwdxb.com
compute:
- hyperthreading: Enabled
  name: worker
  replicas: 0
controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3
metadata:
  name: ocp
//
networking:
  networkType: ncp
  clusterNetwork:
  - cidr: 10.4.0.0/16
    hostPrefix: 23
  machineCIDR: 192.168.25.0/24
  serviceNetwork:
  - 172.30.0.0/16
//
platform:
  vsphere:
    vcenter: ali-vcsa7.vmwdxb.com
    username: administrator@vsphere.local
    password: xxxxxxxx
    datacenter: Ali-vSphere7-DC
    defaultDatastore: OCP-VSAN
    fips: false
pullSecret: 'xxxxx'
sshKey: 'xxxxxx'

Copy NSX-T Operator YAMLs to OpenShift Manifests

Once we generated OpenShift Manifests using “openshift-install create manifests”, we need to do is copy the deploy folder YAML files to the OpenShift Mainfests folder

cp nsx-container-plugin-operator/deploy/* OPENSHIFT-INSTALL-FOLDER/manifests

The rest is standard UPI OpenShift deployment. we can continue deployment using Terraform or by manual VMs deployment as per OpenShift deployment guide. The Operator will deploy apply tags to NSX-T objects and deploy NCP. NCP will take care of provisioning all the needed Networking and Security constructs such L2 Segments, Routers, Load Balancers, IP Pools, Network Policy,…etc.

Thank you for reading.