Components:

Broker (CRD’s)

A broker is a Kubernetes cluster with required CRD’s to store cluster information in its etcd, this cluster only holds cluster resource definitions: clusters.submariner.io and endpoints.submariner.io which defines the other clusters part of submariner. Submariner gateways utilize broker to exchange metadata information between clusters for connection information and updates the local cluster information into the central broker.

Each Submariner enabled cluster joins the broker using service account token and broker-cluster API information. As shown below a broker cluster stores CRD’s with no other Submariner components in ‘’submariner-k8s-broker’ namespace.

Broker Cluster — Custom Resource Definitions

Each object type ‘Cluster’ holds cluster specific Pod and Service subnet information. In IPsec terminology network address of the LAN behind specific Kubernetes cluster (used in IPsec configuration as left and right subnets) which in this case is an overlay network provided by CNI-Flannel.

For the topology shown above, there are four cluster objects (excluding broker):

Broker Cluster — Cluster Objects

Each cluster object contains information on the cluster’s network information (Pod and Service Subnets) and cluster_id (used in IPsec configuration as Left ID and Right ID):

Broker Cluster — Cluster Configuration

Each object type ‘endpoint’ holds public address of the cluster (“left” and “right” declarations in traditional IPsec configuration).

Broker Cluster — Endpoint Objects

Broker Cluster — Endpoint Configuration

Broker also holds other secrets required for cluster membership:

Broker Cluster — Secrets

The datastoresyncer runs as a controller within the leader-elected submariner pod, and is responsible for performing a two-way synchronization between the datastore and local cluster of Submariner CRDs. The datastoresyncer will only push CRD data to the central broker for the local cluster (based on cluster ID), and will sync all data from the broker to the local cluster when the data does not match the local cluster (to prevent circular loops).

All other Kubernetes clusters with Submariner deployed also holds the CRD’s above synced with the broker. These clusters contain a gateway, route-agent and an operator (if Submariner is deployed using subctl).

Submariner — Kubernetes Components

submariner-gateway (DaemonSet)

Gateway pods are run on all the gateway-labeled nodes on host network and a leader election between multiple gateway-labeled nodes is performed to elect an active IPsec endpoint. Upon startup and failure, the submariner-gateway pod that is elected leader will perform a reconciliation process that ensures it is the sole endpoint for this cluster and remote clusters will reconcile the IPsec endpoint to the new endpoint and connections will be re-established accordingly. This entity runs the charon daemon.

Submariner — Gateway Engine

Gateway constantly polls any new cluster information stored as CRD’s from the broker cluster (broker connectivity information-url is provided while creation of the gateway) to configure endpoints (ipsec-peers):

Submariner — Gateway Engine interaction with Broker

As specified, each submariner engine joins the broker using API server token as shown below:

Submariner — Gateway Engine — Broker

submariner-route-agent (DaemonSet)

The submariner-route-agent is deployed as a daemonset (runs on every node) and is aware of the current active gateway on the cluster. Route agent running on the active gateway node creates a VXLAN VTEP interface to which the submariner-route-agent instances running on the other worker nodes connect by establishing a VXLAN tunnel with the VTEP of the active gateway node.

Submariner — Route Agent

Submariner-route-agent also configures routes and programs the necessary iptable rules to enable full connectivity to the remote clusters. The MTU of Submariner VXLAN tunnel is configured based on the MTU of the default interface on the host minus VXLAN overhead.

Submariner — Route Agent — VXLAN

Taking the topology from this scenario required iptables are programmed by Submariner. Each clusters pod and subnet CIDR’s are programmed as shown below:

Submariner — Programmed iptables

All traffic between two clusters will transit between the leader elected (in each cluster) gateway nodes, through ip xfrm rules. Each gateway node has a running Charon daemon which will perform IPsec keying and policy management. xfrm is an IP framework for transforming packets (such as encrypting their payloads). This framework is used to implement the IPsec protocol suite (with the state object operating on the Security Association Database, and the policy object operating on the Security Policy Database).