On-premise kubernetes cluster networking

Purpose of this article is to achive DNS connectivity with on-premise kubernetes cluster which doesn’t have Public IP/Gateway directly.

Overview

If you don’t have public ip on your on-premise cluster then it’s very deficult to expose service to the internet, there is some way to expose using tunnel but there also some limitation. If you want expose your cluster like manage service, then you can follow this article to understand then network behind it.

Network diagram

Required skills

1. Basics of Networking
2. Wiregurd Tunnel or Private Networking with NetBird/Tailscale
3. Kubernetes
4. Helm charts

Networking

Here I am classifing the networks in two parts- 1. External Network and 2. Internal Network.

1. External Network

In this network we have to create an Exit Node and Your DNS server will point to exit node.

a. Exit Node: Exit Node is a normal VPS/VM Instance with minimul configuration which has one fixed public IP address. You have to create an Wiregurd Server on it or you can use any networking solution like NetBird or Tailscale. After that you need to install nginx reverse proxy on it then redirect all the request to your kubernetes master node IP(We will discuss this on later part of this article).

b. DNS: You can use any DNS management system like Cloudflare or your domain provider DNS management system. Then you can assign your Exit Node IP to it. You can use wildcard domain like *.example.com or fixed DNS like example.com or www.example.com.

2. Internal Network

In this network you should use a subnet for this example we are using 20.10.2.0/24. Then connect this internal network through Wireguard or NetBird or Talescale to your Exit Node. After that expose your subnet routing table to your Exit Node.

Lets, create three node kubernetes cluster- Master Node with IP 20.10.2.10/24 and two worker node 20.10.2.11/24 and 20.10.2.12/24.

What need to do in k8s cluster?

You need to setup load balancer inside kubernetes to route your trafic to spacifc application. These are the steps based on hierarchy-

1. MetalLB: Install metal LB load balancer on your cluster:

1

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.5/config/manifests/metallb-native.yaml

2. Nginx Ingress Controller: Install ingress controller on your cluster:

1
2
3
4
5
6
7

kubectl create namespace ingress-controller
helm install nginx-ingress ingress-nginx/ingress-nginx \
  --namespace ingress-controller --create-namespace \
  --set controller.hostNetwork=false \
  --set controller.service.type=LoadBalancer \
  --set controller.service.loadBalancerIP=20.10.2.10 \
  --set controller.replicaCount=1

3. Deployments: Now you are ready to deploy your application:

1
2

kubectl create namespace my-app
kubectl apply -f deployment.yaml -n my-app

3. Servcice: Your service type should be ClusterIP:

1

kubectl apply -f service.yaml -n my-app

3. Ingress: Define your ingress to use ingressClassName as nginx:

Example of ingress.yaml for your application.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app
  namespace: my-app
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  ingressClassName: nginx
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          service:
            name: my-app
            port:
              number: 80
        path: /
        pathType: Prefix

Now deploy your ingress to kubernetes.

1

kubectl apply -f ingress -n my-app

Now you should be able to access your application using your domain through internet.