初始化日志文件： [root@master1 log]# kubeadm init --kubernetes-version=v1.28.2 --image-repository registry.aliyuncs.com/google_containers --v=5 --control-plane-endpoint "192.168.14.214:6443" --upload-certs --service-cidr=10.96.0.0/12 --pod-network-cidr=10.244.0.0/16 I1107 16:01:06.615130 30802 initconfiguration.go:117] detected and using CRI socket: unix:///var/run/containerd/containerd.sock I1107 16:01:06.615547 30802 interface.go:432] Looking for default routes with IPv4 addresses I1107 16:01:06.615563 30802 interface.go:437] Default route transits interface "enp1s0" I1107 16:01:06.615685 30802 interface.go:209] Interface enp1s0 is up I1107 16:01:06.615744 30802 interface.go:257] Interface "enp1s0" has 2 addresses :[192.168.14.214/24 fe80::8e4:7d5b:49d5:5a1a/64]. I1107 16:01:06.615772 30802 interface.go:224] Checking addr 192.168.14.214/24. I1107 16:01:06.615804 30802 interface.go:231] IP found 192.168.14.214 I1107 16:01:06.615829 30802 interface.go:263] Found valid IPv4 address 192.168.14.214 for interface "enp1s0". I1107 16:01:06.615841 30802 interface.go:443] Found active IP 192.168.14.214 I1107 16:01:06.615896 30802 kubelet.go:196] the value of KubeletConfiguration.cgroupDriver is empty; setting it to "systemd" [init] Using Kubernetes version: v1.28.2 [preflight] Running pre-flight checks I1107 16:01:06.620286 30802 checks.go:563] validating Kubernetes and kubeadm version I1107 16:01:06.620312 30802 checks.go:168] validating if the firewall is enabled and active I1107 16:01:06.628057 30802 checks.go:203] validating availability of port 6443 I1107 16:01:06.628308 30802 checks.go:203] validating availability of port 10259 I1107 16:01:06.628353 30802 checks.go:203] validating availability of port 10257 I1107 16:01:06.628394 30802 checks.go:280] validating the existence of file /etc/kubernetes/manifests/kube-apiserver.yaml I1107 16:01:06.628414 30802 checks.go:280] validating the existence of file /etc/kubernetes/manifests/kube-controller-manager.yaml I1107 16:01:06.628427 30802 checks.go:280] validating the existence of file /etc/kubernetes/manifests/kube-scheduler.yaml I1107 16:01:06.628439 30802 checks.go:280] validating the existence of file /etc/kubernetes/manifests/etcd.yaml I1107 16:01:06.628455 30802 checks.go:430] validating if the connectivity type is via proxy or direct I1107 16:01:06.628496 30802 checks.go:469] validating http connectivity to first IP address in the CIDR I1107 16:01:06.628517 30802 checks.go:469] validating http connectivity to first IP address in the CIDR I1107 16:01:06.628531 30802 checks.go:104] validating the container runtime I1107 16:01:06.651081 30802 checks.go:639] validating whether swap is enabled or not I1107 16:01:06.651147 30802 checks.go:370] validating the presence of executable crictl I1107 16:01:06.651182 30802 checks.go:370] validating the presence of executable conntrack I1107 16:01:06.651201 30802 checks.go:370] validating the presence of executable ip I1107 16:01:06.651468 30802 checks.go:370] validating the presence of executable iptables I1107 16:01:06.651493 30802 checks.go:370] validating the presence of executable mount I1107 16:01:06.651516 30802 checks.go:370] validating the presence of executable nsenter I1107 16:01:06.651537 30802 checks.go:370] validating the presence of executable ebtables I1107 16:01:06.651556 30802 checks.go:370] validating the presence of executable ethtool I1107 16:01:06.651574 30802 checks.go:370] validating the presence of executable socat I1107 16:01:06.651594 30802 checks.go:370] validating the presence of executable tc I1107 16:01:06.651613 30802 checks.go:370] validating the presence of executable touch I1107 16:01:06.651637 30802 checks.go:516] running all checks I1107 16:01:06.657342 30802 checks.go:401] checking whether the given node name is valid and reachable using net.LookupHost I1107 16:01:06.657489 30802 checks.go:605] validating kubelet version I1107 16:01:06.699270 30802 checks.go:130] validating if the "kubelet" service is enabled and active I1107 16:01:06.705820 30802 checks.go:203] validating availability of port 10250 I1107 16:01:06.706000 30802 checks.go:329] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables I1107 16:01:06.706039 30802 checks.go:329] validating the contents of file /proc/sys/net/ipv4/ip_forward I1107 16:01:06.706069 30802 checks.go:203] validating availability of port 2379 I1107 16:01:06.706104 30802 checks.go:203] validating availability of port 2380 I1107 16:01:06.706136 30802 checks.go:243] validating the existence and emptiness of directory /var/lib/etcd [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' I1107 16:01:06.706228 30802 checks.go:828] using image pull policy: IfNotPresent I1107 16:01:06.727800 30802 checks.go:846] image exists: registry.aliyuncs.com/google_containers/kube-apiserver:v1.28.2 I1107 16:01:06.749006 30802 checks.go:846] image exists: registry.aliyuncs.com/google_containers/kube-controller-manager:v1.28.2 I1107 16:01:06.770215 30802 checks.go:846] image exists: registry.aliyuncs.com/google_containers/kube-scheduler:v1.28.2 I1107 16:01:06.791257 30802 checks.go:846] image exists: registry.aliyuncs.com/google_containers/kube-proxy:v1.28.2 W1107 16:01:06.812178 30802 checks.go:835] detected that the sandbox image "registry.aliyuncs.com/google_containers/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.aliyuncs.com/google_containers/pause:3.9" as the CRI sandbox image. I1107 16:01:06.833415 30802 checks.go:846] image exists: registry.aliyuncs.com/google_containers/pause:3.9 I1107 16:01:06.854336 30802 checks.go:846] image exists: registry.aliyuncs.com/google_containers/etcd:3.5.9-0 I1107 16:01:06.874799 30802 checks.go:846] image exists: registry.aliyuncs.com/google_containers/coredns:v1.10.1 [certs] Using certificateDir folder "/etc/kubernetes/pki" I1107 16:01:06.874869 30802 certs.go:112] creating a new certificate authority for ca [certs] Generating "ca" certificate and key I1107 16:01:07.082126 30802 certs.go:519] validating certificate period for ca certificate [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master1] and IPs [10.96.0.1 192.168.14.214] [certs] Generating "apiserver-kubelet-client" certificate and key I1107 16:01:07.635288 30802 certs.go:112] creating a new certificate authority for front-proxy-ca [certs] Generating "front-proxy-ca" certificate and key I1107 16:01:07.954870 30802 certs.go:519] validating certificate period for front-proxy-ca certificate [certs] Generating "front-proxy-client" certificate and key I1107 16:01:08.110576 30802 certs.go:112] creating a new certificate authority for etcd-ca [certs] Generating "etcd/ca" certificate and key I1107 16:01:08.817961 30802 certs.go:519] validating certificate period for etcd/ca certificate [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [localhost master1] and IPs [192.168.14.214 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [localhost master1] and IPs [192.168.14.214 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key I1107 16:01:09.813579 30802 certs.go:78] creating new public/private key files for signing service account users [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" I1107 16:01:09.978457 30802 kubeconfig.go:103] creating kubeconfig file for admin.conf [kubeconfig] Writing "admin.conf" kubeconfig file I1107 16:01:10.139104 30802 kubeconfig.go:103] creating kubeconfig file for kubelet.conf [kubeconfig] Writing "kubelet.conf" kubeconfig file I1107 16:01:10.362879 30802 kubeconfig.go:103] creating kubeconfig file for controller-manager.conf [kubeconfig] Writing "controller-manager.conf" kubeconfig file I1107 16:01:10.653254 30802 kubeconfig.go:103] creating kubeconfig file for scheduler.conf [kubeconfig] Writing "scheduler.conf" kubeconfig file [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" I1107 16:01:10.819222 30802 local.go:65] [etcd] wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml" [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" I1107 16:01:10.819263 30802 manifests.go:102] [control-plane] getting StaticPodSpecs I1107 16:01:10.819543 30802 certs.go:519] validating certificate period for CA certificate I1107 16:01:10.819602 30802 manifests.go:128] [control-plane] adding volume "ca-certs" for component "kube-apiserver" I1107 16:01:10.819614 30802 manifests.go:128] [control-plane] adding volume "etc-pki" for component "kube-apiserver" I1107 16:01:10.819630 30802 manifests.go:128] [control-plane] adding volume "k8s-certs" for component "kube-apiserver" I1107 16:01:10.820243 30802 manifests.go:157] [control-plane] wrote static Pod manifest for component "kube-apiserver" to "/etc/kubernetes/manifests/kube-apiserver.yaml" [control-plane] Creating static Pod manifest for "kube-controller-manager" I1107 16:01:10.820266 30802 manifests.go:102] [control-plane] getting StaticPodSpecs I1107 16:01:10.820419 30802 manifests.go:128] [control-plane] adding volume "ca-certs" for component "kube-controller-manager" I1107 16:01:10.820434 30802 manifests.go:128] [control-plane] adding volume "etc-pki" for component "kube-controller-manager" I1107 16:01:10.820443 30802 manifests.go:128] [control-plane] adding volume "flexvolume-dir" for component "kube-controller-manager" I1107 16:01:10.820453 30802 manifests.go:128] [control-plane] adding volume "k8s-certs" for component "kube-controller-manager" I1107 16:01:10.820466 30802 manifests.go:128] [control-plane] adding volume "kubeconfig" for component "kube-controller-manager" I1107 16:01:10.821040 30802 manifests.go:157] [control-plane] wrote static Pod manifest for component "kube-controller-manager" to "/etc/kubernetes/manifests/kube-controller-manager.yaml" [control-plane] Creating static Pod manifest for "kube-scheduler" I1107 16:01:10.821063 30802 manifests.go:102] [control-plane] getting StaticPodSpecs I1107 16:01:10.821210 30802 manifests.go:128] [control-plane] adding volume "kubeconfig" for component "kube-scheduler" I1107 16:01:10.821606 30802 manifests.go:157] [control-plane] wrote static Pod manifest for component "kube-scheduler" to "/etc/kubernetes/manifests/kube-scheduler.yaml" I1107 16:01:10.821628 30802 kubelet.go:67] Stopping the kubelet [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet I1107 16:01:10.994816 30802 waitcontrolplane.go:83] [wait-control-plane] Waiting for the API server to be healthy [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 7.503630 seconds I1107 16:01:18.500191 30802 uploadconfig.go:112] [upload-config] Uploading the kubeadm ClusterConfiguration to a ConfigMap [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace I1107 16:01:18.611046 30802 uploadconfig.go:126] [upload-config] Uploading the kubelet component config to a ConfigMap [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster I1107 16:01:18.680313 30802 uploadconfig.go:131] [upload-config] Preserving the CRISocket information for the control-plane node I1107 16:01:18.680353 30802 patchnode.go:31] [patchnode] Uploading the CRI Socket information "unix:///var/run/containerd/containerd.sock" to the Node API object "master1" as an annotation [upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace [upload-certs] Using certificate key: a9e4a5322f419afd1d4e9b36bc139fd7aee7d4153e54d5c6456980be64745bb3 [mark-control-plane] Marking the node master1 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] [mark-control-plane] Marking the node master1 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule] [bootstrap-token] Using token: jqlj5j.pgec2718h6rz4snh [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace I1107 16:01:20.016364 30802 clusterinfo.go:47] [bootstrap-token] loading admin kubeconfig I1107 16:01:20.016997 30802 clusterinfo.go:58] [bootstrap-token] copying the cluster from admin.conf to the bootstrap kubeconfig I1107 16:01:20.017709 30802 clusterinfo.go:70] [bootstrap-token] creating/updating ConfigMap in kube-public namespace I1107 16:01:20.087076 30802 clusterinfo.go:84] creating the RBAC rules for exposing the cluster-info ConfigMap in the kube-public namespace I1107 16:01:20.112390 30802 kubeletfinalize.go:90] [kubelet-finalize] Assuming that kubelet client certificate rotation is enabled: found "/var/lib/kubelet/pki/kubelet-client-current.pem" [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key I1107 16:01:20.113546 30802 kubeletfinalize.go:134] [kubelet-finalize] Restarting the kubelet to enable client certificate rotation [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of the control-plane node running the following command on each as root: kubeadm join 192.168.14.214:6443 --token jqlj5j.pgec2718h6rz4snh \ --discovery-token-ca-cert-hash sha256:65e8c76b867ad0228c4301eefdd4ae0fe874e05f72280a0dc5204229fdf67420 \ --control-plane --certificate-key a9e4a5322f419afd1d4e9b36bc139fd7aee7d4153e54d5c6456980be64745bb3 Please note that the certificate-key gives access to cluster sensitive data, keep it secret! As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use "kubeadm init phase upload-certs --upload-certs" to reload certs afterward. Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.14.214:6443 --token jqlj5j.pgec2718h6rz4snh \ --discovery-token-ca-cert-hash sha256:65e8c76b867ad0228c4301eefdd4ae0fe874e05f72280a0dc5204229fdf67420