이 글은 '코어 쿠버네티스'의 5장 내용을 실습한 내용입니다.
kind에서 Calico CNI 를 테스트 해본다.
kind 클러스터에 아래 config를 제공해 기본 CNI를 비활성화 하고 클러스터를 생성한다.
# cat kind-Calico-conf.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
disableDefaultCNI: true
podSubnet: 192.168.0.0/16
nodes:
- role: control-plane
- role: workerkind 설정에 대해서는 아래 문서를 참고할 수 있다.
https://kind.sigs.k8s.io/docs/user/configuration/
책이 출간된 이후 apiVersion이 변경되었으므로 kind.sigs.k8s.io가 아닌 kind.x-k8s.io를 사용해야 한다.
# kind create cluster --name=calico --config=./kind-Calico-conf.yaml
ERROR: failed to create cluster: unknown apiVersion: kind.sigs.k8s.io/v1alpha4정상적으로 완료되면 아래와 같이 확인 가능하다. defaultCNI가 disable 되었으므로 아래와 같은 결과가 정상이다.
# kind create cluster --name=calico --config=./kind-Calico-conf.yaml
Creating cluster "calico" ...
✓ Ensuring node image (kindest/node:v1.27.3) 🖼
✓ Preparing nodes 📦 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing StorageClass 💾
✓ Joining worker nodes 🚜
Set kubectl context to "kind-calico"
You can now use your cluster with:
kubectl cluster-info --context kind-calico
Have a nice day! 👋
# kubectl get no
NAME STATUS ROLES AGE VERSION
calico-control-plane NotReady control-plane 40s v1.27.3
calico-worker NotReady <none> 19s v1.27.3
# kubectl get po -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-5d78c9869d-5zkr4 0/1 Pending 0 44s
coredns-5d78c9869d-t2c9x 0/1 Pending 0 44s
etcd-calico-control-plane 1/1 Running 0 57s
kube-apiserver-calico-control-plane 1/1 Running 0 57s
kube-controller-manager-calico-control-plane 1/1 Running 0 59s
kube-proxy-kqsfb 1/1 Running 0 39s
kube-proxy-qqmw4 1/1 Running 0 44s
kube-scheduler-calico-control-plane 1/1 Running 0 57s
아래와 같이 Calico CNI를 설치한다.
먼저 calico operator와 CRD를 생성한다.
# kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.3/manifests/tigera-operator.yaml
namespace/tigera-operator created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpfilters.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created
serviceaccount/tigera-operator created
clusterrole.rbac.authorization.k8s.io/tigera-operator created
clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created
deployment.apps/tigera-operator created위 단계에서는 operator ns와 operator가 생성된다.
# kubectl get po -A -w
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-5d78c9869d-5zkr4 0/1 Pending 0 8m42s
kube-system coredns-5d78c9869d-t2c9x 0/1 Pending 0 ..
tigera-operator tigera-operator-f6bb878c4-x4whq 1/1 Running 0 56s
실제로 calico를 설치한다.
# kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.3/manifests/custom-resources.yaml
installation.operator.tigera.io/default created
apiserver.operator.tigera.io/default created
아래와 같이 노드도 Ready 상태가 되고, calico 파드와 기타 네트워크 연결이 안되서 Pending이던 파드들이 보드 Running 상태가 되었다.
# kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
calico-apiserver calico-apiserver-76cc85cf7f-b2j7p 1/1 Running 0 5m24s
calico-apiserver calico-apiserver-76cc85cf7f-lgnjs 1/1 Running 0 5m24s
calico-system calico-kube-controllers-5f6db5bc7b-ntvhc 1/1 Running 0 9m56s
calico-system calico-node-45w75 1/1 Running 0 9m57s
calico-system calico-node-s8nzv 1/1 Running 0 9m57s
calico-system calico-typha-5bbf9665bd-5mf7t 1/1 Running 0 9m57s
calico-system csi-node-driver-f4958 2/2 Running 0 9m56s
calico-system csi-node-driver-v54cj 2/2 Running 0 9m56s
kube-system coredns-5d78c9869d-5zkr4 1/1 Running 0 21m
kube-system coredns-5d78c9869d-t2c9x 1/1 Running 0 21m
kube-system etcd-calico-control-plane 1/1 Running 0 21m
kube-system kube-apiserver-calico-control-plane 1/1 Running 0 21m
kube-system kube-controller-manager-calico-control-plane 1/1 Running 0 21m
kube-system kube-proxy-kqsfb 1/1 Running 0 21m
kube-system kube-proxy-qqmw4 1/1 Running 0 21m
kube-system kube-scheduler-calico-control-plane 1/1 Running 0 21m
local-path-storage local-path-provisioner-6bc4bddd6b-jhj7s 1/1 Running 0 21m
tigera-operator tigera-operator-f6bb878c4-x4whq 1/1 Running 0 13m
# kubectl get no
NAME STATUS ROLES AGE VERSION
calico-control-plane Ready control-plane 22m v1.27.3
calico-worker Ready <none> 21m v1.27.3책이 출간된 이후로 변경되어 아래 링크를 참고했다. 현재는 operator 방식을 사용하는 것으로 보인다.
https://docs.tigera.io/calico/latest/getting-started/kubernetes/kind
그리고 daemonset로 calico-node 만 있었던 것에서 calico-typha 라는 파드가 추가로 생기는 아키텍처 변화가 있었던 것으로 보인다.
node에서 노드에 필요한 BGP와 IP 경로를 설정하고, Typha에서 API server를 watch 하면서 k8s 리소스와 caclio custom resource의 변화를 바탕으로 node를 업데이트 한다.
https://docs.tigera.io/calico/latest/getting-started/kubernetes/hardway/install-typha
Typha sits between the Kubernetes API server and per-node daemons like Felix and confd (running in `calico/node`). It watches the Kubernetes resources and Calico custom resources used by these daemons, and whenever a resource changes it fans out the update to the daemons.
calico에 대한 추가 테스트는 다시 작성한다.
테스트를 마치면 아래의 명령으로 클러스터를 삭제할 수 있다.
kind delete cluster --name calico