Mastering Certificate Challenges with Ease

Amrutha Paladugu

Introduction:

Kong Hybrid mode represents a groundbreaking approach to API management, merging the control plane (CP) and data plane (DP) capabilities. This mode offers many benefits that empower organizations to achieve enhanced control, security, and performance in their API ecosystem. In this article, we’ll delve into a real-world scenario showcasing the capabilities of Kong Hybrid mode. Specifically, we’ll explore the deployment of Kong Hybrid across Google Kubernetes Engine (GKE) and an On-Premise Kubernetes cluster. We will focus on achieving secure communication, certificate management, workload deployment, and external access. Join us as we navigate this journey through a hybrid API management deployment.

Kong Hybrid architecture with one DP on the cloud and one DP on-premise.

Understanding Certificates:

Mutual Transport Layer Security (mTLS) plays a vital role in Kong’s security features by facilitating secure communication between services. When operating in hybrid mode, the authentication process relies on an mTLS handshake to ensure that communication between CP and DP nodes remains secure, requiring the presentation of certificates. Kong manages certificate key pairs in two distinct modes: Shared and PKI.
In the Shared mode (the default setting), Kong CLI generates a certificate/key pair, and copies of this pair are then distributed across nodes. CP and DP nodes share and utilize this certificate/key pair.

On the other hand, in the PKI mode, certificates signed by a central certificate authority (CA) are provided. Kong performs validation on both ends by verifying if they originate from the same CA. This approach effectively mitigates the risks associated with the transportation of private keys and provides increased security. If one of the DP nodes is compromised, an attacker won’t be able to affect other nodes in the Kong cluster.
In this article, we will use PKI mode and generate three different cert/key pairs for CP and the 2 DPs. All three pairs need to be signed by the same CA discussed above.
Additionally, we will provision a Let’s Encrypt certificate for DP1’s public-facing endpoint and verify encrypted traffic.

Certificate generation:
Step 1: Generate CA cert using OpenSSL:

openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -key ca.key -sha256 -days 365 -out ca.crt

Step2: Generate private key and CSR(certificate signing request) for CP and DPs:

openssl req -new -newkey rsa:2048 -nodes -keyout cp-tls.key -out cp-tls.csr -subj "/CN=*.kong.example.com"
openssl req -new -newkey rsa:2048 -nodes -keyout dp1-tls.key -out dp1-tls.csr -subj "/CN=*.kong.example.com"
openssl req -new -newkey rsa:2048 -nodes -keyout dp2-tls.key -out dp2-tls.csr -subj "/CN=*.127.0.0.1.nip.io"

The CN (Common Name) here would be your wildcard domain.

Step3: Use CA from Step1 to sign the generated CSRs from Step2:

openssl x509 -req -in cp-tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out cp-tls.crt -days 365
openssl x509 -req -in dp1-tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out dp1-tls.crt -days 365
openssl x509 -req -in dp2-tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out dp2-tls.crt -days 365

Step4: Create bundle certs:

cat cp-tls.crt ca.crt > cp-tls-bundle.crt
cat dp1-tls.crt ca.crt > dp1-tls-bundle.crt
cat dp2-tls.crt ca.crt > dp2-tls-bundle.crt

Note: Save these certificates in an accessible directory (certs) for the CP and the DPs

Install CP and DP1 on GKE:

Connect to a GKE cluster and use the next set of commands to install CP and DP1.

Create ns, Create secrets, update helm repo:

sudo snap install helm - classic
helm repo add kong https://charts.konghq.com
helm repo update
kubectl create namespace kong-cp
kubectl create namespace kong-dp1

Create secrets for Kong Enterprise license, Session Config, Admin password, and Postgresql password:

kubectl create secret generic kong-enterprise-license -n kong-cp \
from-file=license=license.json \
dry-run=client -o yaml | kubectl apply -f -
kubectl create secret generic kong-session-config -n kong-cp \
from-literal=portal_session_conf='{"storage":"kong","secret":"super_secret_salt_string","cookie_name":"portal_session","cookie_samesite":"off","cookie_secure":false, "cookie_domain": ".example.com"}' \
from-literal=admin_gui_session_conf='{"cookie_name":"admin_session","cookie_samesite":"off","secret":"super_secret_salt_string","cookie_secure":false,"storage":"kong", "cookie_domain": ".example.com"}'
kubectl create secret generic kong-manager-password -n kong-cp \
from-literal=password=password

kubectl create secret generic kong-cp-postgresql -n kong-cp \
from-literal=host="enterprise-postgresql.kong.svc.cluster.local" \
from-literal=password=password

Note: If you don’t have access to an Enterprise license, you can still go ahead and try the installation but will not be able to access the Admin GUI

Now, create Kubernetes Secrets for the cluster cert and CA cert for ns kong-cp.

kubectl create secret tls kong-cluster-cert -n kong-cp \
cert=./certs/cp-tls-bundle.crt \
key=./certs/cp-tls.key
kubectl create secret generic kong-ca-cert -n kong-cp \
from-file=ca.crt=./certs/ca.crt

We use Helm to install the CP and DPs and use custom values.yaml files.

Update the values-cp.yaml with the below values:

image:
repository: kong/kong-gateway
tag: "3.4"
secretVolumes:
kong-cluster-cert
kong-ca-cert
env:
role: control_plane
cluster_mtls: pki
cluster_ca_cert: /etc/secrets/kong-ca-cert/ca.crt
cluster_cert: /etc/secrets/kong-cluster-cert/tls.crt
cluster_cert_key: /etc/secrets/kong-cluster-cert/tls.key
nginx_http_ssl_protocols: TLSv1.2
admin_gui_ssl_cert: /etc/secrets/kong-cluster-cert/tls.crt
admin_gui_ssl_cert_key: /etc/secrets/kong-cluster-cert/tls.key
admin_ssl_cert: /etc/secrets/kong-cluster-cert/tls.crt
admin_ssl_cert_key: /etc/secrets/kong-cluster-cert/tls.key
portal_gui_ssl_cert: /etc/secrets/kong-cluster-cert/tls.crt
portal_gui_ssl_cert_key: /etc/secrets/kong-cluster-cert/tls.key
portal_api_ssl_cert: /etc/secrets/kong-cluster-cert/tls.crt
portal_api_ssl_cert_key: /etc/secrets/kong-cluster-cert/tls.key
lua_ssl_verify_depth: 3
admin_api_uri: https://admin.kong.example.com
admin_gui_url: https://manager.kong.example.com
portal_auth: basic-auth
portal_api_url: https://portal-api.kong.example.com
portal_gui_url: https://portal-dev.kong.example.com
portal_gui_host: portal-dev.kong.example.com
portal_gui_protocol: https
database: postgres
pg_host: kong-cp-postgresql
pg_database: kong
pg_user: postgres
pg_schema: kong
pg_password:
valueFrom:
secretKeyRef:
name: kong-cp-postgresql
key: password
portal_session_conf:
valueFrom:
secretKeyRef:
name: kong-session-config
key: portal_session_conf
user: kong
password:
valueFrom:
secretKeyRef:
name: kong-manager-password
key: password

admin:
enabled: true
type: ClusterIP
annotations:
konghq.com/protocol: https
labels: {}
http:
enabled: false
tls:
enabled: true
servicePort: 443
containerPort: 8444
parameters:
- http2
ingress:
annotations:
konghq.com/https-redirect-status-code: "301"
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
enabled: true
ingressClassName: kong-dev
hostname: admin.kong.example.com
path: /
tls: kong-cluster-cert

manager:
enabled: true
type: ClusterIP
annotations:
konghq.com/protocol: https
labels: {}
http:
enabled: false
tls:
enabled: true
servicePort: 443
containerPort: 8445
parameters:
- http2
ingress:
annotations:
konghq.com/https-redirect-status-code: "301"
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
enabled: true
ingressClassName: kong-dev
hostname: manager.kong.example.com
path: /
tls: kong-cluster-cert

cluster:
enabled: true
type: LoadBalancer
annotations: {}
http:
enabled: false
tls:
enabled: true
servicePort: 443
containerPort: 8005
ingress:
annotations:
konghq.com/https-redirect-status-code: "301"
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
enabled: true
ingressClassName: kong-dev
hostname: cp.kong.example.com
path: /
tls: kong-cluster-cert

clustertelemetry:
enabled: true
type: LoadBalancer
http:
enabled: false
tls:
enabled: true
servicePort: 443
containerPort: 8006
ingress:
annotations:
konghq.com/https-redirect-status-code: "301"
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
enabled: true
ingressClassName: kong-dev
hostname: tel.kong.example.com
path: /
tls: kong-cluster-cert

proxy:
enabled: false
ingressController:
enabled: true
ingressClass: kong-dev
resources:
limits:
cpu: 200m
memory: 256Mi
requests:
cpu: 100m
memory: 128Mi
installCRDs: false

env:
publish_service: kong-dp/kong-dp-kong-proxy
kong_admin_token:
valueFrom:
secretKeyRef:
name: kong-manager-password
key: password

enterprise:
enabled: true
license_secret: kong-enterprise-license # CHANGEME
vitals:
enabled: false
rbac:
enabled: true
session_conf_secret: kong-session-config
admin_api_auth: basic-auth
portal:
enabled: true
smtp:
enabled: false

portal:
enabled: true
type: ClusterIP
annotations:
konghq.com/protocol: https
labels: {}
http:
enabled: false
tls:
enabled: true
servicePort: 443
containerPort: 8446
ingress:
annotations:
konghq.com/https-redirect-status-code: "301"
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
enabled: true
ingressClassName: kong-dev
hostname: portal-dev.kong.example.com
path: /
tls: kong-cluster-cert

portalapi:
enabled: true
type: ClusterIP
annotations:
konghq.com/protocol: https
labels: {}
http:
enabled: false
tls:
enabled: true
servicePort: 443
containerPort: 8447
ingress:
annotations:
konghq.com/https-redirect-status-code: "301"
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
enabled: true
ingressClassName: kong-dev
hostname: portal-api.kong.example.com
path: /
tls: kong-cluster-cert

postgresql:
enabled: true
auth:
database: kong
username: postgres
existingSecret: kong-cp-postgresql
secretKeys:
adminPasswordKey: password

Let’s try and understand the above values file:
The secrets we created, kong-cluster-cert and kong-ca-cert are mounted as volumes for the SSL certificates and are accessed at the specified path, for example:

cluster_ca_cert: /etc/secrets/kong-ca-cert/ca.crt

The environment variables are for configuring Kong and its components. They control settings related to SSL/TLS, database connections, authentication, and more. Also, they specify the role as “control_plane” and enable cluster mutual TLS (mTLS) using the PKI method. The admin API and GUI URLs, authentication methods, and other configuration parameters are also defined using the env variables.

Admin, Manager, Portal, Portal API, and PostgreSQL:

• These sections configure various components of Kong. All these components are enabled for the CP and would be disabled in the DP values yaml. Their type ( ClusterIP), TLS, and Ingress settings are also defined, including annotations, hostnames, paths, and TLS certificates for these components.
• PostgreSQL settings specify authentication details for connecting to the PostgreSQL database used by Kong.
Enterprise-related settings include the license secret, RBAC configuration, SMTP, and vitals.

Proxy and Ingress Controller:

• The proxy section specifies whether the Kong proxy is enabled. For CP values, this would be disabled as the DP does all the proxying.
• Ingress Controller settings define the resources and environment variables for the Kong Ingress Controller.

Cluster and Cluster Telemetry:

• Cluster and cluster telemetry are essential for Kong Gateway’s Control Plane (CP) and Data Plane (DP) connection. DP instances connect to the CP through the CP cluster, which provides redundancy, load balancing, and configuration synchronization.
• Cluster telemetry plays a crucial role in monitoring the health of both CP and DP components, ensuring they can work together effectively and respond to changes and issues in real time. This setup provides a reliable and resilient CP-DP connection in Kong Gateway deployments.

Install the CP

helm upgrade - install kong-cp kong/kong - namespace kong-cp -f values-cp.yaml

Once the CP is installed, verify that all pods are running and the Cluster and Clusterterlemetry services’ external LoadBalancer IPs are ready.

To install DP in the same cluster, follow the steps below and create license and certificate secrets for DP1.

kubectl create secret generic kong-enterprise-license -n kong-dp1 \
--from-file=license=license.json \
--dry-run=client -o yaml | kubectl apply -f -

kubectl create secret tls kong-cluster-cert -n kong-dp1 \
cert=./certs/dp1-tls-bundle.crt \
key=./certs/dp1-tls.key

kubectl create secret generic kong-ca-cert -n kong-dp1 \
from-file=ca.crt=./certs/ca.crt

To encrypt the oneway traffic to the public-facing end point of DP1, we use Let’s Encrypt CA. Use the below link to generate one for the domain you use.

SSL Certificate Generator: Free letsencrypt SSL in minutes:

A free SSL Certificate Generator. No login is required. Secure your site with a letsencrypt certificate. Includes a…
punchsalad.com

Create a secret for this cert in the kong-dp1 namespace:

kubectl create secret tls kong-tls-cert -n kong-dp1 \
cert=./letsencrypt/ca.crt \
key=./letsencrypt/ca.key

Create a values-dp1.yaml file with the below values:

image:
repository: kong/kong-gateway
tag: "3.4"
secretVolumes:
kong-cluster-cert
kong-ca-cert
kong-tls-cert
env:
role: data_plane
database: "off"
status_listen: 0.0.0.0:8100
cluster_control_plane: cp.kong.example.com:443
cluster_telemetry_endpoint: tel.kong.example.com:443
cluster_mtls: pki
cluster_server_name: cp.kong.example.com
cluster_telemetry_server_name: tel.kong.example.com
lua_ssl_verify_depth: 3
cluster_ca_cert: /etc/secrets/kong-ca-cert/ca.crt
lua_ssl_trusted_certificate: /etc/secrets/kong-cluster-cert/tls.crt
cluster_cert: /etc/secrets/kong-cluster-cert/tls.crt
cluster_cert_key: /etc/secrets/kong-cluster-cert/tls.key
ssl_cert: /etc/secrets/kong-tls-cert/tls.crt
ssl_cert_key: /etc/secrets/kong-tls-cert/tls.key
ingressController:
enabled: false
enterprise:
enabled: true
license_secret: kong-enterprise-license
vitals:
enabled: false
portal:
enabled: false
rbac:
enabled: false
admin:
enabled: false
manager:
enabled: false
portal:
enabled: false
portalapi:
enabled: false
proxy:
enabled: true
type: LoadBalancer
annotations: {}
tls:
enabled: true
servicePort: 443
containerPort: 8443
parameters:
- http2
# stream: {}
ingress:
enabled: true

Please take note of the key differences here from the earlier values-cp yaml. All the Manager, Admin, Portal, Portalapi, and IngressController sections are disabled, while the Proxy section is enabled as type LoadBalancer. As explained earlier, ensuring the correct configuration of cluster and cluster telemetry endpoints is crucial.

Another point worth mentioning is the secret volume for the Let’s Encrypt certificate: kong-tls-cert. This is accessed through the below variables:

ssl_cert: /etc/secrets/kong-tls-cert/tls.crt
ssl_cert_key: /etc/secrets/kong-tls-cert/tls.key

Install DP1:

helm upgrade --install kong-dp1 kong/kong --namespace kong-dp1 -f values-dp1.yaml

Note the LoadBalancer IPs for the Cluster and Clustertelemetry service in the CP and the Proxy service in the DP. Map these values to the domain you intend to use. In our scenario, it would be as follows:

cp.kong.example.com - - - - - - - > 34.136.32.39

cp-tel.kong.example.com - - - - >34.68.58.255

*.kong.example.com - - - - - - - - >34.132.64.159

You should now be able to access Kong Manager (in case you are using an Enterprise license) at manager.kong.example.com, but the browser won’t recognize the OpenSSL CA, and hence it will appear as NotSecure as shown in the below image.

Access Admi GUI and view certificate (Not Secure- openSSL)

Deploy a sample application:

Let’s deploy a sample nodeJS application in the same GKE cluster in the default namespace, try accessing it through the DP endpoint, and see if the Let’s Encrypt certificate is served.

kubectl create deploy --image debianmaster/nodejs-welcome welcome
kubectl expose deployment welcome --type=LoadBalancer --name=welcome --port=8080 --target-port=8080

Deploying a sample nodeJS ‘welcome’ application

Add a service and a route to this application through the Kong manager.

Now, the workload can be accessed through dp1.kong.example.com/welcome. Click on the lock icon🔒present right before the URL and view the certificate. You will notice that the Let’s Encrypt certificate is being served, as shown in the below image.

Access the workload through the DP endpoint and verify the Let’s Encrypt cert.

Install DP2 on local K3s
Connect to a local K3s (lightweight Kubernetes) cluster and install the following command set to install DP2.

Create namespace for DP2:
kubectl create ns kong-dp2

Create secrets for enterprise licenses and certificates:

kubectl create secret generic kong-enterprise-license - from-file=license=./license.json -n kong-dp2 - dry-run=client -o yaml | kubectl apply -f -

kubectl create secret tls kong-cluster-cert -n kong-dp2 \
cert=./certs/dp2-tls-bundle.crt \
key=./certs/dp2-tls.key

kubectl create secret generic kong-ca-cert -n kong-dp2 \
from-file=ca.crt=./certs/ca.crt

Create a values-dp2.yaml file with the below values:

image:
repository: kong/kong-gateway
tag: "3.4"
secretVolumes:
kong-cluster-cert
kong-ca-cert
env:
role: data_plane
cluster_control_plane: cp.kong.example.com:443
cluster_telemetry_endpoint: tel.kong.example.com:443
lua_ssl_trusted_certificate: /etc/secrets/kong-cluster-cert/tls.crt
cluster_mtls: pki
cluster_server_name: cp.kong.example.com
cluster_telemetry_server_name: tel.kong.example.com
cluster_ca_cert: /etc/secrets/kong-ca-cert/ca.crt
cluster_cert: /etc/secrets/kong-cluster-cert/tls.crt
cluster_cert_key: /etc/secrets/kong-cluster-cert/tls.key
database: "off"
audit_log: "on"
headers: "off"
anonymous_reports: "off"
untrusted_lua: "off"
enterprise:
enabled: true
license_secret: kong-enterprise-license
vitals:
enabled: false
portal:
enabled: false
rbac:
enabled: false
admin:
enabled: false
manager:
enabled: false
portal:
enabled: false
portalapi:
enabled: false
ingressController:
enabled: false
proxy:
enabled: true
type: LoadBalancer
annotations: {
konghq.com/protocol: "https"
}
tls:
enabled: true
servicePort: 443
containerPort: 8443
parameters:
http2
ingress:
enabled: true

Install DP2
helm upgrade - install kong-dp2 kong/kong - namespace kong-dp2 -f values-dp2.yaml

Deploy a sample application in the K3s cluster
Let’s deploy a sample nodeJS application in the same K3s cluster in the default namespace and try to access it through the DP2 endpoint and verify.

kubectl create deploy --image debianmaster/nodejs-welcome hello
kubectl expose deployment hello --type=LoadBalancer --name=hello --port=8080 --target-port=8080

Verify deployment of nodeJS ‘hello’ application.

Add a service and a route to this application through the Kong manager.

Now the workload can be accessed through dp2.127.0.0.1.nip.io/hello

Conclusion:

Throughout this article, we embarked on a journey to uncover the transformative power of Kong Hybrid mode within API management. By harmonizing the control plane (CP) and data plane (DP), Kong Hybrid empowers organizations to balance control, security, and performance seamlessly. Through a real-world scenario, we showcased the deployment of Kong Hybrid across Google Kubernetes Engine (GKE) and an On-Premise Kubernetes cluster.

We navigated the intricate landscape of certificate management, emphasizing the pivotal role of Mutual Transport Layer Security (mTLS) and certificates in fortifying Kong’s security architecture. By embracing PKI mode, we ensured secure communication and mitigated the risks inherent in key transportation.

By following our straightforward, step-by-step instructions, we achieved successful deployments of Kong CP and DP1 within the GKE cluster, followed by Kong DP2 in an On-Premise K3s cluster. We highlighted the deployment of sample applications, demonstrated access through DP endpoints, and verified Let’s Encrypt certificates, reinforcing the foundation of a robust and secure API ecosystem.

As we wrap up this journey, remember the importance of resource cleanup to prevent lingering costs. The world of Kong Hybrid mode introduces a revolutionary approach that bridges the gap between Cloud and On-Premise, enabling the creation of APIs that transcend traditional confines. With mastery of this approach, you’ll confidently and innovatively navigate the dynamic realm of modern API management.