Roberto Carratalá

DevOps & Kubernetes enthusiast. MsC in AI/ML & MsC Telco Engineer. Cloud Services Black Belt @ Red Hat

11 minute read

How to ensure that the traffic between my pods on the pod network across my OpenShift cluster it’s encrypted and confidential? How can I enable IPsec to secure my network traffic?

Let’s dig in!

1. Overview

When we have an untrusted network and you want to secure your communications across your cluster for your workloads, encryption services are recommended. Encryption services may also be required for regulatory or compliance reasons (e.g. FIPS compliance).

To achieve this, you can enable IPSec in your OpenShift cluster using the OVN Kubernetes CNI plugin.

IPsec is a protocol suite that enables secure network communications between IP endpoints by providing the following services:

  • Confidentiality
  • Authentication
  • Data Integrity

Broadly used in the Virtual Private Networks (VPNs), IPSec authenticates and encrypts the packets of data to provide secure encrypted communication between two computers over an Internet Protocol network

Also uses cryptographic security services to protect communications over Internet Protocol (IP) networks. It supports network-level peer authentication, data origin authentication, data integrity, data confidentiality (encryption), and replay protection.

2. IPSec Encryption in OpenShift 4 with OVN Kubernetes CNI Plugin

The OVN Kubernetes network plugin uses OVN to instantiate virtual networks for Kubernetes.

These virtual networks use Geneve, a network virtualization overlay encapsulation protocol, to tunnel traffic across the underlay network between Kubernetes Nodes.

OVN tunnel traffic is transported by physical routers and switches. These physical devices could be untrusted (devices in public network) or might be compromised.

Enabling IPsec encryption for this tunnel traffic can prevent the traffic data from being monitored and manipulated, ensuring in that way that OVN data plane (Geneve) traffic between pods on different nodes is confidential, authenticated, and has not been tampered with.

In a nutshell, the purpose of enabling IPsec is to encrypt all traffic between pods on the cluster network when that traffic leaves the node (and correspondingly decrypt traffic that enters the node).

It will not encrypt traffic between pods on the host network, as this traffic does not traverse OVN.

However, as a side effect, this will also encrypt pod traffic that originates from pods on the host network and is destined for the cluster network as this network does traverse OVN

3. Enable IPSec in of OpenShift 4 Cluster

As we described before, with IPsec enabled, all network traffic between nodes on the OVN-Kubernetes Container Network Interface (CNI) cluster network travels through an encrypted tunnel.

IPsec is disabled by default when you install OpenShift 4 clusters. IPsec encryption can be enabled only during cluster installation and cannot be disabled after it is enabled.

Let’s install an OpenShift 4.8 cluster with IPSec enabled (this cluster it’s NOT reachable from outside of a VPN for security purposes :D).

  • First prepare the install-config.yaml:
cat <<EOF > install-config.yaml
apiVersion: v1
baseDomain: rober.lab
compute:
- hyperthreading: Enabled
  name: worker
  replicas: 0
controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3
metadata:
  name: ocp4
networking:
  clusterNetworks:
  - cidr: 10.254.0.0/16
    hostPrefix: 24
  networkType: OVNKubernetes
  serviceNetwork:
  - 172.30.0.0/16
platform:
  none: {}
pullSecret: '$(< ~/.openshift/pull-secret)'
sshKey: '$(< ~/.ssh/id_rsa.pub)'
EOF

NOTE: as you can check the OVN Kubernetes CNI Plugin it’s used in this case.

  • Copy the install-config.yaml and the folder for the installation:
mkdir ocp4
cp -pr install-config.yaml ocp4/
cd ocp4
  • Generate the manifests from the previous install-config.yaml:
openshift-install create manifests
  • Create the cluster network config file inside the manifest folder where the other manifests are located:
cat <<EOF > manifests/cluster-network-03-config.yml
apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  name: cluster
spec:
  defaultNetwork:
    type: OVNKubernetes
    ovnKubernetesConfig:
      ipsecConfig: {}
      mtu: 1400
EOF

IPsec is enabled by updating the Cluster Network Operator configuration during installation.

To enable it the ipsecConfig needs to be configured inside the ovnKubernetesConfig section of a new manifest containing the specifics that will be parsed and used by the CNO for the installation of the cluster.

For more information check the official OpenShift installation advance documentation for network configurations.

With this you can install the OpenShift cluster with the regular procedure as is described in the link before.

Once the OpenShift installation it’s finished, you can check the ovn-ipsec daemonset that manages the daemons (pluto/ovs-monitor-ipsec) responsible for configuring IPsec.

# oc get ds -n openshift-ovn-kubernetes ovn-ipsec
NAME        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
ovn-ipsec   6         6         6       6            6           beta.kubernetes.io/os=linux   3d3h

Because of this daemonset, as you can check there are ipsec pods running in all the nodes of our OCP4 cluster:

# oc get pod -n openshift-ovn-kubernetes -o wide | grep ipsec
ovn-ipsec-4qp86        1/1     Running   0          38m   192.168.7.23   master2.ocp4.rober.lab   <none>           <none>
ovn-ipsec-pk7vh        1/1     Running   0          38m   192.168.7.21   master0.ocp4.rober.lab   <none>           <none>
ovn-ipsec-q4mwj        1/1     Running   0          22m   192.168.7.11   worker0.ocp4.rober.lab   <none>           <none>
ovn-ipsec-trz5m        1/1     Running   0          22m   192.168.7.12   worker1.ocp4.rober.lab   <none>           <none>
ovn-ipsec-vjmw8        1/1     Running   0          38m   192.168.7.22   master1.ocp4.rober.lab   <none>           <none>

Ovn-ipsec contains two daemons (pluto: IPsec IKE daemon, ovs-monitor-ipsec: OVS IPsec daemon):

[root@worker0 ~]# ps aux | grep pluto | tail -n 2 | grep -v grep
root         148  0.0  0.1 380736 14524 ?        Ssl  15:01   0:01 /usr/libexec/ipsec/pluto --leak-detective --config /etc/ipsec.conf --logfile /var/log/openvswitch/libreswan.log

[root@worker0 ~]# ps aux | grep ovs-monitor-ipsec | grep -v grep | tail -n 2
root         172  0.0  0.2 125188 16804 ?        Ss   15:01   0:00 /usr/libexec/platform-python /usr/share/openvswitch/scripts/ovs-monitor-ipsec --pidfile=/var/run/openvswitch/ovs-monitor-ipsec.pid --ike-daemon=libreswan --no-restart-ike-daemon --log-file --detach --monitor unix:/var/run/openvswitch/db.sock
root         173  0.0  0.2 125188 18632 ?        S    15:01   0:02 /usr/libexec/platform-python /usr/share/openvswitch/scripts/ovs-monitor-ipsec --pidfile=/var/run/openvswitch/ovs-monitor-ipsec.pid --ike-daemon=libreswan --no-restart-ike-daemon --log-file --detach --monitor unix:/var/run/openvswitch/db.sock

In a nutshell, ovs-monitor-ipsec reads OVS tables and configures pluto, and pluto configures Linux IPsec via XFRM in the OpenShift cluster.

4. Analysing IPSec configuration in Openshift cluster deployment

The status of IPsec can be inspected by running the OVS commands in the ovn-ipsec container. For example to show the ipsec tunnels configured in our cluster:

oc -n openshift-ovn-kubernetes exec -it ovn-ipsec-q4mwj -c ovn-ipsec -- bash
[root@worker0 ~]#

[root@worker0 ~]# ovs-appctl -t ovs-monitor-ipsec tunnels/show | grep CONFIGURED
Interface name: ovn-ab2a7c-0 v1 (CONFIGURED)
Interface name: ovn-04e0df-0 v1 (CONFIGURED)
Interface name: ovn-91fa72-0 v1 (CONFIGURED)
Interface name: ovn-9469f8-0 v1 (CONFIGURED)

If we can pay attention to one of the interfaces used in the ipsec tunneling, we can check tons of useful information:

[root@worker0 ~]# ovs-appctl -t ovs-monitor-ipsec tunnels/show
Interface name: ovn-ab2a7c-0 v1 (CONFIGURED)
  Tunnel Type:    geneve
  Local IP:       %defaultroute
  Remote IP:      192.168.7.23
  Address Family: IPv4
  SKB mark:       None
  Local cert:     /etc/openvswitch/keys/ipsec-cert.pem
  Local name:     2dc40cb7-845e-4326-9d1d-340f9961d76c
  Local key:      /etc/openvswitch/keys/ipsec-privkey.pem
  Remote cert:    None
  Remote name:    ab2a7cc3-15a3-420b-a81f-d066c38b1c02
  CA cert:        /etc/openvswitch/keys/ipsec-cacert.pem
  PSK:            None
  Ofport:         1
  CFM state:      Disabled
Kernel policies installed:
  src 192.168.7.11/32 dst 192.168.7.23/32 proto udp dport 6081
  src 192.168.7.11/32 dst 192.168.7.23/32 proto udp dport 6081
  src 192.168.7.11/32 dst 192.168.7.23/32 proto udp sport 6081
  src 192.168.7.11/32 dst 192.168.7.23/32 proto udp sport 6081
Kernel security associations installed:
  sel src 192.168.7.23/32 dst 192.168.7.11/32 proto udp sport 6081
  sel src 192.168.7.11/32 dst 192.168.7.23/32 proto udp dport 6081
  sel src 192.168.7.23/32 dst 192.168.7.11/32 proto udp sport 6081
  sel src 192.168.7.11/32 dst 192.168.7.23/32 proto udp dport 6081
  sel src 192.168.7.23/32 dst 192.168.7.11/32 proto udp dport 6081
  sel src 192.168.7.11/32 dst 192.168.7.23/32 proto udp sport 6081
IPsec connections that are active:
  000 #8: "ovn-ab2a7c-0-in-1" esp.30da143@192.168.7.23 esp.9b00b718@192.168.7.11 Traffic: ESPin=8MB ESPout=0B! ESPmax=0B
  000 #15: "ovn-ab2a7c-0-out-1" esp.c7a82264@192.168.7.23 esp.bf5d5971@192.168.7.11 Traffic: ESPin=0B ESPout=3MB! ESPmax=0B

in this case, we’re checking the ipsec connections that are active in this specific interface, among other useful information like the remote IP, the certs, the IKE specs, etc

Managed internally, the Cluster Network Operator creates a self-signed CA certificate and then handles certificate signing requests from each node when they are booted. Each signed cert is shared with the other nodes between which secure communication is allowed.

The status of the ipsec can also be checked with the ipsec/status option in the ovs-appctl command:

[root@worker0 ~]# ovs-appctl -t ovs-monitor-ipsec ipsec/status
{'ovn-04e0df-0': {'ovn-04e0df-0-in-1': '000 #4: "ovn-04e0df-0-in-1" esp.adf6cbe@192.168.7.21 esp.79a9c45@192.168.7.11 Traffic: ESPin=13MB ESPout=0B! ESPmax=0B', 'ovn-04e0df-0-out-1': '000 #13: "ovn-04e0df-0-out-1" esp.5b4ed2ae@192.168.7.21 esp.e4452c0a@192.168.7.11 Traffic: ESPin=0B ESPout=5MB! ESPmax=0B'}, 'ovn-91fa72-0': {'ovn-91fa72-0-in-1': '000 #2: "ovn-91fa72-0-in-1" esp.790d7cb5@192.168.7.22 esp.ad252497@192.168.7.11 Traffic: ESPin=20MB ESPout=0B! ESPmax=0B', 'ovn-91fa72-0-out-1': '000 #11: "ovn-91fa72-0-out-1" esp.375fb81c@192.168.7.22 esp.50e8f4f4@192.168.7.11 Traffic: ESPin=0B ESPout=2193MB! ESPmax=0B'}, 'ovn-9469f8-0': {'ovn-9469f8-0-in-1': '000 #20: "ovn-9469f8-0-in-1" esp.7441c038@192.168.7.12 esp.1fdc5612@192.168.7.11 Traffic: ESPin=24MB ESPout=0B! ESPmax=0B', 'ovn-9469f8-0-out-1': '000 #18: "ovn-9469f8-0-out-1" esp.bdc12ddd@192.168.7.12 esp.55222499@192.168.7.11 Traffic: ESPin=0B ESPout=127MB! ESPmax=0B'}, 'ovn-ab2a7c-0': {'ovn-ab2a7c-0-in-1': '000 #8: "ovn-ab2a7c-0-in-1" esp.30da143@192.168.7.23 esp.9b00b718@192.168.7.11 Traffic: ESPin=9MB ESPout=0B! ESPmax=0B', 'ovn-ab2a7c-0-out-1': '000 #15: "ovn-ab2a7c-0-out-1" esp.c7a82264@192.168.7.23 esp.bf5d5971@192.168.7.11 Traffic: ESPin=0B ESPout=3MB! ESPmax=0B'}}

We can get more information about the ipsec status with a typical libreswan command will also run in this container:

[root@worker0 ~]# ipsec status
000 using kernel interface: xfrm
000
000 interface lo UDP [::1]:500
000 interface lo UDP 127.0.0.1:4500
000 interface lo UDP 127.0.0.1:500
000 interface br-ex UDP 192.168.7.11:4500
000 interface br-ex UDP 192.168.7.11:500
000 interface ovn-k8s-mp0 UDP 10.254.3.2:4500
000 interface ovn-k8s-mp0 UDP 10.254.3.2:500
....
000 #8: "ovn-ab2a7c-0-in-1":500 STATE_V2_ESTABLISHED_CHILD_SA (IPsec SA established); EVENT_SA_REKEY in 25217s; newest IPSEC; eroute owner; isakmp#7; idle;
000 #8: "ovn-ab2a7c-0-in-1" esp.30da143@192.168.7.23 esp.9b00b718@192.168.7.11 Traffic: ESPin=9MB ESPout=0B! ESPmax=0B
000 #9: "ovn-ab2a7c-0-out-1":500 STATE_V2_ESTABLISHED_CHILD_SA (IPsec SA established); EVENT_SA_REKEY in 25482s; isakmp#7; idle;
000 #9: "ovn-ab2a7c-0-out-1" esp.dec7ae77@192.168.7.23 esp.8941d9a5@192.168.7.11 Traffic: ESPin=0B ESPout=0B! ESPmax=0B
000 #15: "ovn-ab2a7c-0-out-1":500 STATE_V2_ESTABLISHED_CHILD_SA (IPsec SA established); EVENT_SA_REKEY in 25902s; newest IPSEC; eroute owner; isakmp#7; idle;
000 #15: "ovn-ab2a7c-0-out-1" esp.c7a82264@192.168.7.23 esp.bf5d5971@192.168.7.11 Traffic: ESPin=0B ESPout=3MB! ESPmax=0B
000
000 Bare Shunt list:
000

NOTE: Libreswan is a free software implementation of the most widely supported and standardized VPN protocol using “IPsec” and the Internet Key Exchange (“IKE”), and it’s used in the “pluto” component that it’s a Libreswan IKE Daemon.

4.1 Certificates in IPSec

The Cluster Network Operator (CNO) generates a self-signed X.509 certificate authority (CA) that is used by IPsec for encryption. Certificate signing requests (CSRs) from each node are automatically fulfilled by the CNO.

The CA is valid for 10 years. The individual node certificates are valid for 5 years and are automatically rotated after 4 1/2 years elapse.

Local certificates and keys used by the IPSec can be found in the ovn-ipsec pod:

# oc -n openshift-ovn-kubernetes exec -it ovn-ipsec-q4mwj -c ovn-ipsec -- bash
[root@worker0 ~]# ovs-vsctl get open . other_config:ca_cert
"/etc/openvswitch/keys/ipsec-cacert.pem"

[root@worker0 ~]# ovs-vsctl get open . other_config:certificate
"/etc/openvswitch/keys/ipsec-cert.pem"

[root@worker0 ~]# ovs-vsctl get open . other_config:private_key
"/etc/openvswitch/keys/ipsec-privkey.pem"

[root@worker0 ~]# ls -al /etc/openvswitch/keys
total 24
drwxr-xr-x. 2 root root   98 Jul 29 15:01 .
drwxr-xr-x. 3 root root   18 Jul 29 15:01 ..
-rw-------. 1 root root 4392 Jul 29 15:01 ipsec-cacert.pem
-rw-------. 1 root root 4353 Jul 29 15:01 ipsec-cert.pem
-rw-------. 1 root root 1679 Jul 29 15:01 ipsec-privkey.pem
-rw-------. 1 root root 3697 Jul 29 15:01 ipsec-req.pem

To check the cert and the keys, and to see if are configured correctly you can list them with ovs-vsctl list open:

[root@worker0 ~]# ovs-vsctl list open .
_uuid               : 2e86cfd2-3d2f-4190-a910-b7995ffdd97b
bridges             : [b2971eed-d6a0-490d-a6f6-ddcb746bb764, c3a5ef44-66a0-4f42-968d-d0630a962ff3]
cur_cfg             : 64
datapath_types      : [netdev, system]
datapaths           : {}
db_version          : "8.2.0"
dpdk_initialized    : false
dpdk_version        : "DPDK 20.11.1"
external_ids        : {hostname=worker0.ocp4.rober.lab, ovn-bridge-mappings="physnet:br-ex", ovn-encap-ip="192.168.7.11", ovn-encap-type=geneve, ovn-monitor-all="true", ovn-openflow-probe-interval="180", ovn-remote="ssl:192.168.7.21:9642,ssl:192.168.7.22:9642,ssl:192.168.7.23:9642", ovn-remote-probe-interval="30000", rundir="/var/run/openvswitch", system-id="2dc40cb7-845e-4326-9d1d-340f9961d76c"}
iface_types         : [bareudp, erspan, geneve, gre, gtpu, internal, ip6erspan, ip6gre, lisp, patch, stt, system, tap, vxlan]
manager_options     : []
next_cfg            : 64
other_config        : {ca_cert="/etc/openvswitch/keys/ipsec-cacert.pem", certificate="/etc/openvswitch/keys/ipsec-cert.pem", private_key="/etc/openvswitch/keys/ipsec-privkey.pem"}
ovs_version         : "2.15.1"
ssl                 : 1321d39c-b29e-4e0d-809e-0676347a2849
statistics          : {}
system_type         : rhcos
system_version      : "4.8"

As you can check this command shows also interesting information about the versions used, the keys used, bridges and datapaths, etc

5. Traffic Encryption using IPSec in OpenShift

With IPSec enabled, all Geneve traffic is encrypted with AES_GCM_16_256 using IPsec transport mode. The AES_GCM_16_256 encryption used can only be enabled at installation time (FIPS compliance)

The logs in the ovn-ipsec container can also have useful information about this encryption and can be examined at:

[root@worker0 ~]# tail -f /var/log/openvswitch/libreswan.log
Jul 29 15:02:06.832185: "ovn-9469f8-0-in-1" #20: sent CREATE_CHILD_SA request for new IPsec SA
Jul 29 15:02:06.838331: "ovn-9469f8-0-in-1" #20: negotiated connection [192.168.7.11-192.168.7.11:6081-6081 17] -> [192.168.7.12-192.168.7.12:0-65535 17]
Jul 29 15:02:06.838351: "ovn-9469f8-0-in-1" #20: IPsec SA established transport mode {ESP=>0x7441c038 <0x1fdc5612 xfrm=AES_GCM_16_256-NONE-MODP2048 NATOA=none NATD=none DPD=passive}

5.1 Traffic Encryption Analysis with IPSec Enabled

To check that effectively the traffic between the nodes it’s encrypted we will perform a tcpdump in an OpenShift Cluster that have IPSec enabled.

When IPSec it’s enabled there should be ‘ESP’ traffic flowing between nodes and no “Geneve” traffic.

  • Let’s check in one OpenShift node the traffic flow with the tcpdump command:
[root@helper ~]# oc debug node/worker0.ocp4.rober.lab -- tcpdump esp
Starting pod/worker0ocp4roberlab-debug ...
To use host binaries, run `chroot /host`
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-ex, link-type EN10MB (Ethernet), capture size 262144 bytes
15:49:35.997554 IP worker0.ocp4.rober.lab > master2.ocp4.rober.lab: ESP(spi=0xc7a82264,seq=0x3e7a), length 1472
15:49:35.997618 IP worker0.ocp4.rober.lab > master2.ocp4.rober.lab: ESP(spi=0xc7a82264,seq=0x3e7b), length 432
15:49:35.997847 IP worker1.ocp4.rober.lab > worker0.ocp4.rober.lab: ESP(spi=0x1fdc5612,seq=0xf590), length 132
15:49:35.997996 IP worker0.ocp4.rober.lab > worker1.ocp4.rober.lab: ESP(spi=0xbdc12ddd,seq=0x2161b), length 132
15:49:35.998317 IP worker1.ocp4.rober.lab > worker0.ocp4.rober.lab: ESP(spi=0x1fdc5612,seq=0xf591), length 644
15:49:35.998402 IP worker0.ocp4.rober.lab > worker1.ocp4.rober.lab: ESP(spi=0xbdc12ddd,seq=0x2161c), length 124
15:49:35.998425 IP master2.ocp4.rober.lab > worker0.ocp4.rober.lab: ESP(spi=0x9b00b718,seq=0x4bfb), length 136

If you pay attention to the output of the tcpdump, we can see the esp in each communication between the nodes of our cluster (worker / masters).

ESP is the acronym of Encapsulating Security Payload. It provides origin authenticity through source authentication, data integrity through hash functions and confidentiality through encryption protection for IP packets. ESP also supports encryption-only and authentication-only configurations, but using encryption without authentication is strongly discouraged because it is insecure.

5.2 Traffic Analysis with IPSec NOT Enabled

In another cluster that have not enabled the IPSec perform the same tcpdump (but without filtering by ESP):

oc debug node/compute-0 -- tcpdump

15:51:13.515446 IP compute-0.compute.local.4194 > master-2.compute.local.geneve: Geneve, Flags [C], vni 0x4, options [8 bytes]: IP compute-0.38062 > 10.129.0.40.xmltec-xmlmail: Flags [S], seq 474331479, win 27200, options [mss 1360,sackOK,TS val 3574026533 ecr 0,nop,wscale 7], length 0
15:51:13.515651 IP master-2.compute.local.29614 > compute-0.compute.local.geneve: Geneve, Flags [C], vni 0xd, options [8 bytes]: IP 10.129.0.40.xmltec-xmlmail > compute-0.38062: Flags [S.], seq 944206554, ack 474331480, win 26960, options [mss 1360,sackOK,TS val 1969320340 ecr 3574026533,nop,wscale 7], length 0
15:51:13.515756 IP compute-0.compute.local.4194 > master-2.compute.local.geneve: Geneve, Flags [C], vni 0x4, options [8 bytes]: IP compute-0.38062 > 10.129.0.40.xmltec-xmlmail: Flags [P.], seq 1:518, ack 1, win 213, options [nop,nop,TS val 3574026533 ecr 1969320340], length 517

As we can see the traffic shows Geneve, that is not encrypted by default so we can see the traffic flows and we can not ensure that the traffic is not tampered.

And with that, we completed our analysis of IPSec in an OpenShift cluster!

NOTE: Opinions expressed in this blog are my own and do not necessarily reflect that of the company I work for.

Stay tuned and happy OpenShifting!

You may also enjoy