In this blog I will go through NSX-T 3.0 and OpenShift 4.4 integration using the new NSX-T Operator for OpenShift. The Blog is based on personal testing for the NSX-T Openshift Operator published here and it is not validated by both vendors.
The test was done on vSphere 7 with VSAN storage.
We can start deploying OpenShift using the deployment guide on vSphere.
NSX-T will provide all the Networking and Security functionality needed for OpenShift such as connectivity, IPAM, NetworkPolicy, Load Balancing, OpenShift Routes/Ingress, and traffic visibility.
This blog assumes good level of understating for both OpenShift and NSX-T. The steps will go as below,
1. Configure DNS and Load Balancing
2. Prepare NSX-T Components
3. Configure NSX Container Plugin (NCP) and publish NCP and Operator Images
4. Install OpenShift with NSX-T Operator
Configure DNS and Load Balancing
The first step is a standard for OpenShift Deployments. Nothing changes for DNS requirements.
for Load Balancing, NSX-T Load Balancer could be used. We will need to create two Active Monitors, two Server Pools, and two Load Balancing VIPs with the same IP address. One for TCP-6443, and another one for TCP-22623. We can use NSX-T Dynamic Group Membership to configure the Server Pools as shown in below example.
Both api and api-int DNS entries should point to the VIPs IP address.
Prepare NSX-T Components
There are multiple components we need to create
- T0 Gateway for North-South connectivity with Static Routes or BGP
- T1 Gateway for OpenShift Nodes connectivity
- L2 Segment for OpenShift Nodes connectivity
- DHCP Server for OpenShift Bootstrap process
Depending on NCP configuration. The OpenShift Pods may be connectected to a separate T1-GW than the OpenShift Nodes.
Publish NCP and Operator Images and configure NCP
There are two images that we will need.
- NCP Image
- NSX-T OpenShift Operator Image
The NCP Image could be downloaded from https://my.vmware.com/web/vmware/downloads (needs my-vmware account). While the Operator one could be built as described here https://github.com/vmware/nsx-container-plugin-operator
Once both images are available, they will need to be published in some container registry as we will need to pint to them later.
Now lets clone the Operator repo to configure NCP and the Operator
We will need to edit two files under the deploy folder.
- Edit deploy/operator.yaml to point to NCP and Operator images
26 image: OPERATOR_IMAGE 37 value: "NCP_IMAGE"
- Edit deploy/configmap.yaml with your environment variables. Below are the minimum that needs modification
117 nsx_api_managers = NSXT-MANAGER-IP 119 nsx_api_user = NSXT-USER 121 nsx_api_password = NSXT-PASSWORD 256 external_ip_pools = EXTERNAL_IP_CIDR 263 tier0_gateway = T0-GW-NAME 281 overlay_tz = OVERLAY-TRANSPORT-ZONE-UUID 348 edge_cluster = EDGE-CLUSTER-UUID 473 apiserver_host_ip = api-int.OPENSHIFT-CLUSTER.BASE-DOMAIN
There are couple of deviation from the standard OpenShift UPI deployment.
- Modify install-config.yaml
- Copy the operator “deploy” folder to OpenShift Manifests
When deploying OpenShift, the first step is to create the install-config.yaml. we need to add “networking” section highlighted so OpenShift use NCP as a CNI instead of using openshift-sdn.
apiVersion: v1 baseDomain: vmwdxb.com compute: - hyperthreading: Enabled name: worker replicas: 0 controlPlane: hyperthreading: Enabled name: master replicas: 3 metadata: name: ocp // networking: networkType: ncp clusterNetwork: - cidr: 10.4.0.0/16 hostPrefix: 23 machineCIDR: 192.168.25.0/24 serviceNetwork: - 172.30.0.0/16 // platform: vsphere: vcenter: ali-vcsa7.vmwdxb.com username: email@example.com password: xxxxxxxx datacenter: Ali-vSphere7-DC defaultDatastore: OCP-VSAN fips: false pullSecret: 'xxxxx' sshKey: 'xxxxxx'
Copy NSX-T Operator YAMLs to OpenShift Manifests
Once we generated OpenShift Manifests using “openshift-install create manifests”, we need to do is copy the deploy folder YAML files to the OpenShift Mainfests folder
cp nsx-container-plugin-operator/deploy/* OPENSHIFT-INSTALL-FOLDER/manifests
The rest is standard UPI OpenShift deployment. we can continue deployment using Terraform or by manual VMs deployment as per OpenShift deployment guide. The Operator will deploy apply tags to NSX-T objects and deploy NCP. NCP will take care of provisioning all the needed Networking and Security constructs such L2 Segments, Routers, Load Balancers, IP Pools, Network Policy,…etc.
Thank you for reading.