用MetalLB将Kubernetes的LoadBalancer服务部署看看
你好。
我是Class Act公司基础设施业务部的大塚。
本文讨论在使用microk8s构建的Kubernetes集群环境中部署MetalLB和LoadBalancer服务，然后通过负载均衡器，尝试从Web浏览器访问Pod中的容器。
环境
k8s集群的环境如下所示。有3个节点。具有192.168.2.30 IP的节点是Master节点。其他具有31、32 IP的节点作为worker节点工作。
在k8s集群中没有运行pod和service等（确切地说，正在运行但不必担心）。
這個指令的日誌顯示了沒有啟動任何東西並且有三個節點。
服務正在運行，不需要擔心。同時在其他命名空間也啟動了各種東西，但不需要擔心，這些是默認創建的。
現在讓我們立即開始建立。
root@k8s-master:~/yaml# kubectl get all -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 24h <none>
root@k8s-master:~/yaml# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-worker01 Ready <none> 7h32m v1.26.4 192.168.2.31 <none> Ubuntu 22.04.2 LTS 5.15.0-71-generic containerd://1.6.15
k8s-master Ready <none> 24h v1.26.4 192.168.2.30 <none> Ubuntu 22.04.2 LTS 5.15.0-71-generic containerd://1.6.15
k8s-worker02 Ready <none> 7h31m v1.26.4 192.168.2.32 <none> Ubuntu 22.04.2 LTS 5.15.0-71-generic containerd://1.6.15
在中文中用語是一种常见的表达方式。
在构建之前，我们将简要总结一下这次出现的k8s相关术语。
部署
部署是以声明方式进行Pod和ReplicaSet的更新的工具。
通过描述所需的状态来定义部署，部署控制器会以受控速度将实际状态改变为所需的状态。您可以定义部署来创建新的ReplicaSet，或删除现有的部署并将其资源全部采用新的部署。
副本集
ReplicaSet的目的是稳定地维护一组正在运行的副本Pod，以确保指定数量的相同Pod的可用性经常被使用。
MetalLB 是一个用于 Kubernetes 的开源负载均衡器。
MetalLB 是一种适用于裸金属环境下的 Kubernetes 外部负载均衡器实现。它是由谷歌开发的简单负载均衡器，并提供两个功能：为 LoadBalancer 类型的服务分配公开用 IP 地址（外部 IP），以及传播与外部 IP 相关的路由信息。
负载均衡器服务
以下是该段的汉语释义：在集群之外的客户端可以使用它来访问Pod组。与NodePort不同，它可以使用外部负载均衡器来公开Service，不需要考虑节点的IP地址或端口号，而是可以使用配置在外部负载均衡器上的虚拟IP地址进行访问。负载均衡器注册的负载均衡目标取决于使用的CNI。
使用Deployment对象部署Pod。
以下是准备好的yaml文件。
将deployment的名称设置为nginx-httpd-deployment，并将永久展开的pod数量设置为3。
pod的内容是从之前的qiita文章中复用的。换句话说，我们使用的是一个pod内同时运行nginx容器和apache2容器的内容。
root@k8s-master:~/yaml# cat web-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-httpd-deployment
labels:
app: web-app-deployment
spec:
replicas: 3
selector:
matchLabels:
app: web-app
template:
metadata:
labels:
app: web-app
spec:
containers:
- name: web-httpd
image: shotaohtsuka/my-httpd-image
ports:
- name: web-httpd
containerPort: 90
protocol: TCP
- name: web-nginx
image: nginx
ports:
- name: web-nginx
containerPort: 80
protocol: TCP
根据此yaml文件执行部署。
从”get all”命令的输出结果中，我们可以看出每个节点都创建了一个pod。此外，输出还包括了deployment和其中的replicas。
root@k8s-master:~/yaml# kubectl create -f web-deployment.yaml
deployment.apps/nginx-httpd-deployment created
root@k8s-master:~/yaml# kubectl get all -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/nginx-httpd-deployment-759b9b5f-vq8rp 2/2 Running 0 7s 10.1.235.209 k8s-master <none> <none>
pod/nginx-httpd-deployment-759b9b5f-rsp86 2/2 Running 0 7s 10.1.79.68 k8s-worker01 <none> <none>
pod/nginx-httpd-deployment-759b9b5f-h8k67 2/2 Running 0 7s 10.1.69.197 k8s-worker02 <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 24h <none>
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/nginx-httpd-deployment 3/3 3 3 7s web-httpd,web-nginx shotaohtsuka/my-httpd-image,nginx app=web-app
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
replicaset.apps/nginx-httpd-deployment-759b9b5f 3 3 3 7s web-httpd,web-nginx shotaohtsuka/my-httpd-image,nginx app=web-app,pod-template-hash=759b9b5f
部署负载均衡器服务
为了将上述部署的pod集群公开给外部，我们将部署LoadBalancer服务。我们为此准备的yaml文件如下所示。
首先，我們使用名為lb-service-httpd-nginx的LoadBalancer服務。LB在8080端口和9090端口上進行監聽，節點在30080端口和30090端口上進行監聽。Pod在80端口和90端口上進行監聽。
當通信以LB的IP經過8080端口到達時，將被轉發到每個節點的30080端口上進行通信；當通信到達每個節點的30080端口時，則NodePort會將通信轉發到每個Pod的80端口上，最終連接到Pod內的nginx。
同樣地，當通信以LB的IP經過9090端口到達時，將被轉發到每個節點的30090端口上進行通信；當通信到達每個節點的30090端口時，則NodePort會將通信轉發到每個Pod的90端口上，最終連接到Pod內的apache2。
由於我個人有些混亂，稍後會將其轉化為圖像來解釋。
root@k8s-master:~/yaml# cat deployment-service-httpd-nginx.yaml
apiVersion: v1
kind: Service
metadata:
name: lb-service-httpd-nginx
spec:
type: LoadBalancer
selector:
app: web-app
ports:
- name: nginx-port
port: 8080
targetPort: 80
nodePort: 30080
- name: httpd-port
port: 9090
targetPort: 90
nodePort: 30090
我将部署这个。
部署本身没有问题，但是service/lb-service-httpd-nginx的EXTERNAL-IP是pending状态，我希望在这里有一个用于负载均衡的IP。
为此，我将启用MetalLB。
root@k8s-master:~/yaml# kubectl get all -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/nginx-httpd-deployment-759b9b5f-vq8rp 2/2 Running 0 16m 10.1.235.209 k8s-master <none> <none>
pod/nginx-httpd-deployment-759b9b5f-rsp86 2/2 Running 0 16m 10.1.79.68 k8s-worker01 <none> <none>
pod/nginx-httpd-deployment-759b9b5f-h8k67 2/2 Running 0 16m 10.1.69.197 k8s-worker02 <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 24h <none>
service/lb-service-httpd-nginx LoadBalancer 10.152.183.197 <pending> 8080:30080/TCP,9090:30090/TCP 6s app=web-app
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/nginx-httpd-deployment 3/3 3 3 16m web-httpd,web-nginx shotaohtsuka/my-httpd-image,nginx app=web-app
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
replicaset.apps/nginx-httpd-deployment-759b9b5f 3 3 3 16m web-httpd,web-nginx shotaohtsuka/my-httpd-image,nginx app=web-app,pod-template-hash=759b9b5f
启用MetalLB。
请输入每个IP地址范围，以逗号分隔（例如，’10.64.140.43-10.64.140.49,192.168.0.105-192.168.0.111’）：应该使用与k8s节点不在同一网络范围内的IP地址。本次为了负载均衡器，已保留第四个八位字段为35-39的IP地址。
root@k8s-master:~/yaml# microk8s status
microk8s is running
high-availability: no
datastore master nodes: 192.168.2.30:19001
datastore standby nodes: none
addons:
enabled:
dashboard # (core) The Kubernetes dashboard
dns # (core) CoreDNS
ha-cluster # (core) Configure high availability on the current node
helm # (core) Helm - the package manager for Kubernetes
helm3 # (core) Helm 3 - the package manager for Kubernetes
metrics-server # (core) K8s Metrics Server for API access to service metrics
disabled:
cert-manager # (core) Cloud native certificate management
community # (core) The community addons repository
gpu # (core) Automatic enablement of Nvidia CUDA
host-access # (core) Allow Pods connecting to Host services smoothly
hostpath-storage # (core) Storage class; allocates storage from host directory
ingress # (core) Ingress controller for external access
kube-ovn # (core) An advanced network fabric for Kubernetes
mayastor # (core) OpenEBS MayaStor
metallb # (core) Loadbalancer for your Kubernetes cluster
minio # (core) MinIO object storage
observability # (core) A lightweight observability stack for logs, traces and metrics
prometheus # (core) Prometheus operator for monitoring and logging
rbac # (core) Role-Based Access Control for authorisation
registry # (core) Private image registry exposed on localhost:32000
storage # (core) Alias to hostpath-storage add-on, deprecated
root@k8s-master:~/yaml# microk8s enable metallb
Infer repository core for addon metallb
Enabling MetalLB
Enter each IP address range delimited by comma (e.g. '10.64.140.43-10.64.140.49,192.168.0.105-192.168.0.111'): 192.168.2.35-192.168.2.39
Applying Metallb manifest
customresourcedefinition.apiextensions.k8s.io/addresspools.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bfdprofiles.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bgpadvertisements.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bgppeers.metallb.io created
customresourcedefinition.apiextensions.k8s.io/communities.metallb.io created
customresourcedefinition.apiextensions.k8s.io/ipaddresspools.metallb.io created
customresourcedefinition.apiextensions.k8s.io/l2advertisements.metallb.io created
namespace/metallb-system created
serviceaccount/controller created
serviceaccount/speaker created
clusterrole.rbac.authorization.k8s.io/metallb-system:controller unchanged
clusterrole.rbac.authorization.k8s.io/metallb-system:speaker unchanged
role.rbac.authorization.k8s.io/controller created
role.rbac.authorization.k8s.io/pod-lister created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller unchanged
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker unchanged
rolebinding.rbac.authorization.k8s.io/controller created
secret/webhook-server-cert created
service/webhook-service created
rolebinding.rbac.authorization.k8s.io/pod-lister created
daemonset.apps/speaker created
deployment.apps/controller created
validatingwebhookconfiguration.admissionregistration.k8s.io/validating-webhook-configuration configured
Waiting for Metallb controller to be ready.
deployment.apps/controller condition met
ipaddresspool.metallb.io/default-addresspool created
l2advertisement.metallb.io/default-advertise-all-pools created
MetalLB is enabled
root@k8s-master:~/yaml# microk8s status
microk8s is running
high-availability: no
datastore master nodes: 192.168.2.30:19001
datastore standby nodes: none
addons:
enabled:
dashboard # (core) The Kubernetes dashboard
dns # (core) CoreDNS
ha-cluster # (core) Configure high availability on the current node
helm # (core) Helm - the package manager for Kubernetes
helm3 # (core) Helm 3 - the package manager for Kubernetes
metallb # (core) Loadbalancer for your Kubernetes cluster
metrics-server # (core) K8s Metrics Server for API access to service metrics
disabled:
cert-manager # (core) Cloud native certificate management
community # (core) The community addons repository
gpu # (core) Automatic enablement of Nvidia CUDA
host-access # (core) Allow Pods connecting to Host services smoothly
hostpath-storage # (core) Storage class; allocates storage from host directory
ingress # (core) Ingress controller for external access
kube-ovn # (core) An advanced network fabric for Kubernetes
mayastor # (core) OpenEBS MayaStor
minio # (core) MinIO object storage
observability # (core) A lightweight observability stack for logs, traces and metrics
prometheus # (core) Prometheus operator for monitoring and logging
rbac # (core) Role-Based Access Control for authorisation
registry # (core) Private image registry exposed on localhost:32000
storage # (core) Alias to hostpath-storage add-on, deprecated
再次确认EXTERNAL-IP。
可以确认分配了192.168.2.35。
root@k8s-master:~/yaml# kubectl get service -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 24h <none>
lb-service-httpd-nginx LoadBalancer 10.152.183.197 192.168.2.35 8080:30080/TCP,9090:30090/TCP 5m40s app=web-app
将通信流程转化为图像的结果如下：
蓝色箭头表示与nginx容器的通信路径，红色箭头表示与apache2容器的通信路径。
当想要连接到pod并确认其内容时，需要指定LoadBalancer的IP地址为192.168.2.35和端口号进行访问。
直接使用Web浏览器访问Pod容器中的内容。
自言自语：【推测】为什么需要负载均衡器？为什么不能使用NodePort？
但是，在这种情况下，例如如果worker01节点死亡了，如果nginx和worker01之间没有通过心跳等确认其生存状态，那么在向已死亡的节点分发负载时，存在无法获取数据的可能性。
我觉得需要考虑一些生存确认等实现方面的问题，这可能会相当困难。使用LoadBalancer服务可以很好地处理这些事情…。
