使用 kubeadm 搭建 v1.17.4 版本 Kubernetes 集群

一、环境准备

1.1实验环境

角色 IP地址 主机名 docker版本 硬件配置 系统 内核
master 10.0.0.130 k8s-master 19.03.4 2c2g CentOS7.6 3.10.0-957.el7.x86_64
node1 10.0.0.131 k8s-node1 19.03.4 2c1g CentOS7.6 3.10.0-957.el7.x86_64
node2 10.0.0.132 k8s-node2 19.03.4 2c1g CentOS7.6 3.10.0-957.el7.x86_64

1.2每个节点配置host信息

cat >> /etc/hosts <<EOF
10.0.0.130 k8s-master
10.0.0.131 k8s-node1
10.0.0.132 k8s-node2
EOF

1.3禁用防火墙和selinux

//禁用防火墙
systemctl stop firewalld && systemctl disable firewalld

//禁用selinux
#临时修改
setenforce 0

#永久修改,重启服务器后生效
sed -i '7s/enforcing/disabled/' /etc/selinux/config

1.4创建/etc/sysctl.d/k8s.conf文件,添加如下内容

//向文件中写入以下内容
cat >/etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

//执行以下命令生效
modprobe br_netfilter && sysctl -p /etc/sysctl.d/k8s.conf

1.5安装ipvs(可选)

脚本创建了的/etc/sysconfig/modules/ipvs.modules文件,保证在节点重启后能自动加载所需模块。使用lsmod | grep -e ip_vs -e nf_conntrack_ipv4命令查看是否已经正确加载所需的内核模块

//向文件中写入以下内容
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF

//修改权限以及查看是否已经正确加载所需的内核模块
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

nf_conntrack_ipv4      15053  0 
nf_defrag_ipv4         12729  1 nf_conntrack_ipv4
ip_vs_sh               12688  0 
ip_vs_wrr              12697  0 
ip_vs_rr               12600  0 
ip_vs                 145497  6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack          133095  2 ip_vs,nf_conntrack_ipv4
libcrc32c              12644  3 xfs,ip_vs,nf_conntrack

安装ipset和ipvsadm(便于查看 ipvs 的代理规则)

yum -y install ipset ipvsadm

1.6同步服务器时间

//安装chrony
yum -y install chrony

//修改同步服务器地址为阿里云
sed -i.bak '3,6d' /etc/chrony.conf && sed -i '3cserver ntp1.aliyun.com iburst' \
/etc/chrony.conf

//启动chronyd及加入开机自启
systemctl start chronyd && systemctl enable chronyd

//查看同步结果
chronyc sources

210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 120.25.115.20                 2   6    37    13   +194us[+6131us] +/-   33ms

1.7关闭swap分区

修改/etc/fstab文件,注释掉 SWAP 的自动挂载,使用free -m确认 swap 已经关闭

//手动关闭swap
swapoff -a

//修改fstab文件,注释swap自动挂载
sed -i '/^\/dev\/mapper\/centos-swap/c#/dev/mapper/centos-swap swap                    swap    defaults        0 0' /etc/fstab

//查看swap是否关闭
free -m

total        used        free      shared  buff/cache   available
Mem:           1994         682         612           9         699        1086
Swap:             0           0           0

swappiness 参数调整,修改/etc/sysctl.d/k8s.conf添加下面一行

cat >>/etc/sysctl.d/k8s.conf <<EOF
vm.swappiness=0
EOF

//使配置生效
sysctl -p /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness = 0

1.8安装docker19.03.4

1.卸载旧版本
yum remove -y docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-selinux \
docker-engine-selinux \
docker-engine

2.安装依赖包
yum install -y yum-utils device-mapper-persistent-data lvm2

3.添加阿里云yum源
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

4.查看可用版本
yum list docker-ce --showduplicates | sort -r

已加载插件:fastestmirror, langpacks
可安装的软件包
 * updates: mirrors.aliyun.com
Loading mirror speeds from cached hostfile
 * extras: mirrors.aliyun.com
docker-ce.x86_64            3:19.03.5-3.el7                     docker-ce-stable
docker-ce.x86_64            3:19.03.4-3.el7                     docker-ce-stable
。。。。。。
docker-ce.x86_64            3:18.09.9-3.el7                     docker-ce-stable
docker-ce.x86_64            3:18.09.8-3.el7                     docker-ce-stable
docker-ce.x86_64            3:18.09.7-3.el7                     docker-ce-stable
docker-ce.x86_64            3:18.09.6-3.el7                     docker-ce-stable
。。。。。。


5.安装docker19.03.4
yum install -y docker-ce-19.03.4 docker-ce-cli-19.03.4 containerd.io

6.启动docker并设置开机自启
systemctl enable docker && systemctl start docker

7.配置阿里云docker镜像加速
cat > /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://gqk8w9va.mirror.aliyuncs.com"]
}
EOF

8.配置完后重启docker
systemctl restart docker

9.查看加速
docker info
找到Registry Mirrors一行
Registry Mirrors:
 https://gqk8w9va.mirror.aliyuncs.com/

10.查看docker版本
docker version

Client: Docker Engine - Community
 Version:           19.03.4
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        9013bf583a
 Built:             Fri Oct 18 15:52:22 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.4
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       9013bf583a
  Built:            Fri Oct 18 15:50:54 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

1.9安装Kubeadm

需要科学上网

cat >/etc/yum.repos.d/kubernetes.repo<<EOF
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
        https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

使用阿里云yum源

cat >/etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
        http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

安装 kubeadm、kubelet、kubectl(阿里云yum源会随官方更新最新版,因此指定版本)

yum -y install kubelet-1.17.4 kubeadm-1.17.4 kubectl-1.17.4 

//查看版本
kubeadm version

kubeadm version: &version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T21:01:11Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

设置kubelet开机自启

⚠️此时kubelet是无法启动的,因为只有完成master的kubeadm init 的操作,kubelet才能正常启动

systemctl enable kubelet

设置k8s命令自动补全

yum -y install bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc

1.10修改docker Cgroup Driver为systemd

#修改docker Cgroup Driver为systemd/usr/lib/systemd/system/docker.service文件中的这一行 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

修改为 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd

如果不修改,在添加 worker 节点时可能会碰到如下错误
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". 
Please follow the guide at https://kubernetes.io/docs/setup/cri/


//使用如下命令修改  
sed -i.bak "s#^ExecStart=/usr/bin/dockerd.*#ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd#g" /usr/lib/systemd/system/docker.service

//重启docker
systemctl daemon-reload && systemctl restart docker

到此,基本环境安装完成!!!

二、初始化集群

2.1master节点操作,配置 kubeadm 初始化文件

  • 方法一
    • 命令初始化
kubeadm init --apiserver-advertise-address=10.0.0.130 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.17.4 --service-cidr=10.96.0.0/16 --pod-network-cidr=10.20.0.0/16

--apiserver-advertise-address=    #master节点IP
--image-repository registry.aliyuncs.com/google_containers #镜像仓库
--kubernetes-version v1.17.4    #k8s版本
--service-cidr=10.96.0.0/16        #service IP网段
--pod-network-cidr=10.20.0.0/16    #pod IP网段,后续网络插件会用到
  • 方法二
    • 文件初始化
cat > kubeadm.yaml <<EOF
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.0.0.130  # apiserver 节点内网IP
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: k8s-master   # master IP地址
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS  # dns类型
etcd:
  local:
    dataDir: /var/lib/etcd
#遗下两个仓库2选1      
#imageRepository: gcr.azk8s.cn/google_containers
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.17.4  # k8s版本
networking:
  dnsDomain: cluster.local
  podSubnet: 10.20.0.0/16
  serviceSubnet: 10.96.0.0/16
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs  # kube-proxy 模式
EOF

2.2初始化master

kubeadm init --config kubeadm.yaml

完整输出结果
W0319 22:59:56.631407   12367 validation.go:28] Cannot validate kubelet config - no validator is available
W0319 22:59:56.631444   12367 validation.go:28] Cannot validate kube-proxy config - no validator is available
[init] Using Kubernetes version: v1.17.4
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.0.0.130]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [10.0.0.130 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [10.0.0.130 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0319 23:01:45.692056   12367 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0319 23:01:45.692762   12367 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 16.502774 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.17" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.0.0.130:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:27b37276c4451b54c99106f73cb38c3b0bb6e2e178c142808e9ff00300ac7eaf
#kubeadm init --config kubeadm.yaml初始化master下载的镜像
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy                v1.17.4             6dec7cfde1e5        6 days ago          116MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver            v1.17.4             2e1ba57fe95a        6 days ago          171MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager   v1.17.4             7f997fcf3e94        6 days ago          161MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler            v1.17.4             5db16c1c7aff        6 days ago          94.4MB
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns                   1.6.5               70f311871ae1        4 months ago        41.6MB
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd                      3.4.3-0             303ce5db0e90        4 months ago        288MB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause                     3.1                 da86e6ba6ca1        2 years ago         742kB


#启动的容器
5220dac932a8        6dec7cfde1e5                                                    "/usr/local/bin/kube…"   2 minutes ago       Up 2 minutes                                 k8s_kube-proxy_kube-proxy-jlwfd_kube-system_5af2149d-dc6c-4a6b-a99b-fb0af734d1fb_0
20a9003d1a55        registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1   "/pause"                 2 minutes ago       Up 2 minutes                                 k8s_POD_kube-proxy-jlwfd_kube-system_5af2149d-dc6c-4a6b-a99b-fb0af734d1fb_0
48c86b9adbf1        7f997fcf3e94                                                    "kube-controller-man…"   3 minutes ago       Up 3 minutes                                 k8s_kube-controller-manager_kube-controller-manager-k8s-master_kube-system_c06ae2244175d69b82dd8327536a3f47_0
beed6b3671bd        5db16c1c7aff                                                    "kube-scheduler --au…"   3 minutes ago       Up 3 minutes                                 k8s_kube-scheduler_kube-scheduler-k8s-master_kube-system_0621ae8690c69d1d72f746bc2de0667e_0
fe47bd324d10        2e1ba57fe95a                                                    "kube-apiserver --ad…"   3 minutes ago       Up 3 minutes                                 k8s_kube-apiserver_kube-apiserver-k8s-master_kube-system_ffa9ef4a4e85500fbc6d35b5a88efb43_0
cfbf4d126eae        303ce5db0e90                                                    "etcd --advertise-cl…"   3 minutes ago       Up 3 minutes                                 k8s_etcd_etcd-k8s-master_kube-system_ba1d164d5064efe1d13e5645859a6dec_0
aff7b239f5ef        registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1   "/pause"                 3 minutes ago       Up 3 minutes                                 k8s_POD_kube-scheduler-k8s-master_kube-system_0621ae8690c69d1d72f746bc2de0667e_0
1266cf8da7d9        registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1   "/pause"                 3 minutes ago       Up 3 minutes                                 k8s_POD_kube-controller-manager-k8s-master_kube-system_c06ae2244175d69b82dd8327536a3f47_0
7415d07612a9        registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1   "/pause"                 3 minutes ago       Up 3 minutes                                 k8s_POD_kube-apiserver-k8s-master_kube-system_ffa9ef4a4e85500fbc6d35b5a88efb43_0
fef33ef8bb46        registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1   "/pause"                 3 minutes ago       Up 3 minutes                                 k8s_POD_etcd-k8s-master_kube-system_ba1d164d5064efe1d13e5645859a6dec_0

拷贝 kubeconfig 文件

//这里的路径为/root
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

2.3master添加节点

node1和node2相同操作

将master节点上的$HOME/.kube/config 文件拷贝到node节点对应的文件中

1.创建目录,这里的路径为/root
mkdir -p $HOME/.kube 

2.把master节点上的config文件拷贝到node1和node2的$HOME/.kube
scp k8s-master:~/.kube/config $HOME/.kube

3.修改权限
chown $(id -u):$(id -g) $HOME/.kube/config

将node1和node2加入到集群中

这里需要用到2.2中初始化master最后生成的token和sha256值

kubeadm join 10.0.0.130:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:27b37276c4451b54c99106f73cb38c3b0bb6e2e178c142808e9ff00300ac7eaf 


输出结果  
W0319 23:06:51.048079    4137 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

将node节点加入到k8s集群后,node节点会运行kube-proxy

#下载的镜像
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy   v1.17.4             6dec7cfde1e5        6 days ago          116MB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause        3.1                 da86e6ba6ca1        2 years ago         742kB

#启动的容器
f8dff6a9caf5        registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy   "/usr/local/bin/kube…"   About a minute ago   Up About a minute                       k8s_kube-proxy_kube-proxy-g8lfq_kube-system_72e60b6a-0023-4d25-b8e0-73e98bb4b1ea_0
64752d1e78b3        registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1    "/pause"                 2 minutes ago        Up 2 minutes                            k8s_POD_kube-proxy-g8lfq_kube-system_72e60b6a-0023-4d25-b8e0-73e98bb4b1ea_0

如果忘记了token和sha256值,可以在master节点使用如下命令查看

//查看token
kubeadm token list

TOKEN                     TTL         EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS
abcdef.0123456789abcdef   23h         2020-03-20T23:02:03+08:00   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token


//查看sha256
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

27b37276c4451b54c99106f73cb38c3b0bb6e2e178c142808e9ff00300ac7eaf


//同时查看token和sha256
kubeadm token create --print-join-command

W0319 23:07:44.720182   16201 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0319 23:07:44.720215   16201 validation.go:28] Cannot validate kubelet config - no validator is available
kubeadm join 10.0.0.130:6443 --token 36t7ur.rtigrd344emr5urd     --discovery-token-ca-cert-hash sha256:27b37276c4451b54c99106f73cb38c3b0bb6e2e178c142808e9ff00300ac7eaf

master节点查看node,发现状态都是NotReady,因为还没有安装网络插件,这里我们安装calio官方插件文档

kubectl get nodes
NAME         STATUS     ROLES    AGE     VERSION
k8s-master   NotReady   master   19m     v1.15.3
k8s-node1    NotReady   <none>   4m10s   v1.15.3
k8s-node2    NotReady   <none>   4m3s    v1.15.3

2.4master节点安装网络插件calio

//下载文件
wget https://kuboard.cn/install-script/calico/calico-3.13.1.yaml  

https://docs.projectcalico.org


//修改完成后安装calico网络插件
kubectl apply -f calico-3.13.1.yaml

//安装完成后稍等一会查看pods状态
kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-788d6b9876-4lls7   1/1     Running   0          5m58s
calico-node-jfn5g                          1/1     Running   0          5m59s
calico-node-k6f84                          1/1     Running   0          5m59s
calico-node-vm4m6                          1/1     Running   0          5m59s
coredns-7f9c544f75-5p2mf                   1/1     Running   0          30m
coredns-7f9c544f75-prdlb                   1/1     Running   0          30m
etcd-k8s-master                            1/1     Running   0          30m
kube-apiserver-k8s-master                  1/1     Running   0          30m
kube-controller-manager-k8s-master         1/1     Running   0          30m
kube-proxy-g8lfq                           1/1     Running   0          25m
kube-proxy-jlwfd                           1/1     Running   0          30m
kube-proxy-x8twc                           1/1     Running   0          25m
kube-scheduler-k8s-master                  1/1     Running   0          30m


//查看node状态
kubectl get nodes
NAME         STATUS   ROLES    AGE   VERSION
k8s-master   Ready    master   30m   v1.17.4
k8s-node1    Ready    <none>   25m   v1.17.4
k8s-node2    Ready    <none>   25m   v1.17.4


#kubectl apply -f calico-3.13.1.yaml安装网络插件calico下载的镜像
calico/node                                                                   v3.13.1             2e5029b93d4a        6 days ago          260MB
calico/pod2daemon-flexvol                                                     v3.13.1             e8c600448aae        6 days ago          111MB
calico/cni                                                                    v3.13.1             6912ec2cfae6        6 days ago          207MB

#启动的容器
70d8a967cc69        calico/node                                                     "start_runit"            2 minutes ago       Up 2 minutes                                        k8s_calico-node_calico-node-jfn5g_kube-system_b86bd096-addf-4665-8a72-ec7c9e4a97f9_0
f9830c017734        calico/pod2daemon-flexvol                                       "/usr/local/bin/flex…"   4 minutes ago       Exited (0) 4 minutes ago                            k8s_flexvol-driver_calico-node-jfn5g_kube-system_b86bd096-addf-4665-8a72-ec7c9e4a97f9_0
8fe868ba10d4        6912ec2cfae6                                                    "/install-cni.sh"        5 minutes ago       Exited (0) 5 minutes ago                            k8s_install-cni_calico-node-jfn5g_kube-system_b86bd096-addf-4665-8a72-ec7c9e4a97f9_0
ae71689904ca        calico/cni                                                      "/opt/cni/bin/calico…"   5 minutes ago       Exited (0) 5 minutes ago                            k8s_upgrade-ipam_calico-node-jfn5g_kube-system_b86bd096-addf-4665-8a72-ec7c9e4a97f9_0

node节点

#都有的镜像
calico/node                                                      
calico/pod2daemon-flexvol                                        
calico/cni            

#calico/kube-controllers 、registry.cn-hangzhou.aliyuncs.com/google_containers/coredns会随机在一个node上

2.5安装Dashboard

下载文件及修改内容

这里查看dashboard对应的k8s版本

//下载文件  v2.0.0-rc3是中文版本,beta8是英文版本
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml

wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc3/aio/deploy/recommended.yaml  

//修改Service为NodePort类型
42行下增加一行
nodePort: 30001

44行下增加一行
type: NodePort


//原先内容
spec:
  ports:
    - port: 443
      targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard


//修改后内容
spec:
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30001   #增加,指定nodeport端口
  selector:
    k8s-app: kubernetes-dashboard
  type: NodePort        #增加,修改类型为nodeport

部署dashboard

#kubectl apply -f recommended.yaml


#下载的镜像
kubernetesui/dashboard                                                        v2.0.0-rc3          4a0a1cf1b459        6 weeks ago         126MB

#启动的容器
62a551c0a0e6        kubernetesui/dashboard   "/dashboard --insecu…"   9 seconds ago       Up 8 seconds                            k8s_kubernetes-dashboard_kubernetes-dashboard-7867cbccbb-z9vzj_kubernetes-dashboard_de95d4c3-6234-4f7a-8d80-d326c52b65c0_0

查看dashboard的运行状态及外网访问端口

//查看dashboard运行状态
kubectl get pods -n kubernetes-dashboard -l k8s-app=kubernetes-dashboard

NAME                                    READY   STATUS    RESTARTS   AGE
kubernetes-dashboard-7867cbccbb-z9vzj   1/1     Running   0          50s



//查看dashboard外网访问端口
kubectl get svc -n kubernetes-dashboard -l k8s-app=kubernetes-dashboard

NAME                   TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
kubernetes-dashboard   NodePort   10.96.222.222   <none>        443:30001/TCP   82s

通过上边的30001端口访问dashboard,注意是https

⚠️k8s1.16.3这个版本中,使用的dashboard版本是2.0.0-beta8,只有火狐浏览器可以访问,其余浏览器都不能访问,会报错如下

iShot2020-03-1722.09.59

只有火狐浏览器可以!

然后创建一个具有全局所有权限的用户来登录Dashboard

//编辑admin.yaml文件
cat > admin.yaml <<EOF
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: admin
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: admin
  namespace: kube-system

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
EOF

//直接创建
kubectl apply -f admin.yaml

//查看token
kubectl get secret -n kube-system|grep admin-token

admin-token-vfmmm                                kubernetes.io/service-account-token   3      7s


//获取base64解码后的字符串,注意需要用到上边命令查看到的token,会生成很长的一串字符串
kubectl get secret admin-token-j7sfh -o jsonpath={.data.token} -n kube-system |base64 -d

#直接用这条命令搞定
kubectl get secret `kubectl get secret -n kube-system|grep admin-token|awk '{print $1}'` -o jsonpath={.data.token} -n kube-system |base64 -d && echo

然后粘贴上述生成的base64字符串登陆dashboard,在登陆页面选择令牌一项

2

登陆后的首界面

1

2.6安装kuboard(可选)

kuboard是Kubernetes 的一款图形化管理界面

安装kuboard

kubectl apply -f https://kuboard.cn/install-script/kuboard.yaml
kubectl apply -f https://addons.kuboard.cn/metrics-server/0.3.6/metrics-server.yaml

查看运行状态

#kubectl get pods -l k8s.eip.work/name=kuboard -n kube-system
NAME                       READY   STATUS    RESTARTS   AGE
kuboard-756d46c4d4-tvhjq   1/1     Running   0          40s

获取token

//获取管理员token
kubectl -n kube-system get secret $(kubectl -n kube-system get secret | grep kuboard-user | awk '{print $1}') -o go-template='{{.data.token}}' | base64 -d && echo


//获取只读token
kubectl -n kube-system get secret $(kubectl -n kube-system get secret | grep kuboard-viewer | awk '{print $1}') -o go-template='{{.data.token}}' | base64 -d && echo

访问kuboard

Kuboard Service 使用了 NodePort 的方式暴露服务,NodePort 为 32567

k8s集群的任意一个节点都可以访问到

iShot2020-03-1722.09.59a

kuboard下载的镜像和启动的容器

#启动的容器
11171ab07891        eipwork/kuboard     "/entrypoint.sh"    3 minutes ago       Up 3 minutes                            k8s_kuboard_kuboard-756d46c4d4-jfx4t_kube-system_8f509c5b-fbee-40b0-8d74-9f2b8485639a_0d

#下载的镜像,本文中为1.0.8.2版本
eipwork/kuboard                                                               latest              c6d652bbdf90        About an hour ago   180MB

到此,使用kubeadm安装k8s 1.17.4完成!!!

获取令牌命令

kubectl get secret `kubectl get secret -n kube-system|grep admin-token|awk '{print $1}'` -o jsonpath={.data.token} -n kube-system |base64 -d && echo

1.17.4master启动的容器及下载的镜像

#启动的容器
11171ab07891        eipwork/kuboard                                                 "/entrypoint.sh"         4 minutes ago       Up 4 minutes                                       k8s_kuboard_kuboard-756d46c4d4-jfx4t_kube-system_8f509c5b-fbee-40b0-8d74-9f2b8485639a_0
f8e5d82cf8ee        registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1   "/pause"                 6 minutes ago       Up 6 minutes                                       k8s_POD_kuboard-756d46c4d4-jfx4t_kube-system_8f509c5b-fbee-40b0-8d74-9f2b8485639a_0
62a551c0a0e6        kubernetesui/dashboard                                          "/dashboard --insecu…"   10 hours ago        Up 10 hours                                        k8s_kubernetes-dashboard_kubernetes-dashboard-7867cbccbb-z9vzj_kubernetes-dashboard_de95d4c3-6234-4f7a-8d80-d326c52b65c0_0
40ad4724f6f1        registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1   "/pause"                 10 hours ago        Up 10 hours                                        k8s_POD_kubernetes-dashboard-7867cbccbb-z9vzj_kubernetes-dashboard_de95d4c3-6234-4f7a-8d80-d326c52b65c0_0
70d8a967cc69        calico/node                                                     "start_runit"            10 hours ago        Up 10 hours                                        k8s_calico-node_calico-node-jfn5g_kube-system_b86bd096-addf-4665-8a72-ec7c9e4a97f9_0
f9830c017734        calico/pod2daemon-flexvol                                       "/usr/local/bin/flex…"   10 hours ago        Exited (0) 10 hours ago                            k8s_flexvol-driver_calico-node-jfn5g_kube-system_b86bd096-addf-4665-8a72-ec7c9e4a97f9_0
8fe868ba10d4        6912ec2cfae6                                                    "/install-cni.sh"        10 hours ago        Exited (0) 10 hours ago                            k8s_install-cni_calico-node-jfn5g_kube-system_b86bd096-addf-4665-8a72-ec7c9e4a97f9_0
ae71689904ca        calico/cni                                                      "/opt/cni/bin/calico…"   10 hours ago        Exited (0) 10 hours ago                            k8s_upgrade-ipam_calico-node-jfn5g_kube-system_b86bd096-addf-4665-8a72-ec7c9e4a97f9_0
cd04477d7393        registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1   "/pause"                 10 hours ago        Up 10 hours                                        k8s_POD_calico-node-jfn5g_kube-system_b86bd096-addf-4665-8a72-ec7c9e4a97f9_0
5220dac932a8        6dec7cfde1e5                                                    "/usr/local/bin/kube…"   10 hours ago        Up 10 hours                                        k8s_kube-proxy_kube-proxy-jlwfd_kube-system_5af2149d-dc6c-4a6b-a99b-fb0af734d1fb_0
20a9003d1a55        registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1   "/pause"                 10 hours ago        Up 10 hours                                        k8s_POD_kube-proxy-jlwfd_kube-system_5af2149d-dc6c-4a6b-a99b-fb0af734d1fb_0
48c86b9adbf1        7f997fcf3e94                                                    "kube-controller-man…"   10 hours ago        Up 10 hours                                        k8s_kube-controller-manager_kube-controller-manager-k8s-master_kube-system_c06ae2244175d69b82dd8327536a3f47_0
beed6b3671bd        5db16c1c7aff                                                    "kube-scheduler --au…"   10 hours ago        Up 10 hours                                        k8s_kube-scheduler_kube-scheduler-k8s-master_kube-system_0621ae8690c69d1d72f746bc2de0667e_0
fe47bd324d10        2e1ba57fe95a                                                    "kube-apiserver --ad…"   10 hours ago        Up 10 hours                                        k8s_kube-apiserver_kube-apiserver-k8s-master_kube-system_ffa9ef4a4e85500fbc6d35b5a88efb43_0
cfbf4d126eae        303ce5db0e90                                                    "etcd --advertise-cl…"   10 hours ago        Up 10 hours                                        k8s_etcd_etcd-k8s-master_kube-system_ba1d164d5064efe1d13e5645859a6dec_0
aff7b239f5ef        registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1   "/pause"                 10 hours ago        Up 10 hours                                        k8s_POD_kube-scheduler-k8s-master_kube-system_0621ae8690c69d1d72f746bc2de0667e_0
1266cf8da7d9        registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1   "/pause"                 10 hours ago        Up 10 hours                                        k8s_POD_kube-controller-manager-k8s-master_kube-system_c06ae2244175d69b82dd8327536a3f47_0
7415d07612a9        registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1   "/pause"                 10 hours ago        Up 10 hours                                        k8s_POD_kube-apiserver-k8s-master_kube-system_ffa9ef4a4e85500fbc6d35b5a88efb43_0
fef33ef8bb46        registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1   "/pause"                 10 hours ago        Up 10 hours                                        k8s_POD_etcd-k8s-master_kube-system_ba1d164d5064efe1d13e5645859a6dec_0


#下载的镜像
eipwork/kuboard                                                               latest              c6d652bbdf90        2 days ago          180MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy                v1.17.4             6dec7cfde1e5        7 days ago          116MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver            v1.17.4             2e1ba57fe95a        7 days ago          171MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager   v1.17.4             7f997fcf3e94        7 days ago          161MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler            v1.17.4             5db16c1c7aff        7 days ago          94.4MB
calico/node                                                                   v3.13.1             2e5029b93d4a        7 days ago          260MB
calico/pod2daemon-flexvol                                                     v3.13.1             e8c600448aae        7 days ago          111MB
calico/cni                                                                    v3.13.1             6912ec2cfae6        7 days ago          207MB
kubernetesui/dashboard                                                        v2.0.0-rc3          4a0a1cf1b459        7 weeks ago         126MB
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns                   1.6.5               70f311871ae1        4 months ago        41.6MB
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd                      3.4.3-0             303ce5db0e90        4 months ago        288MB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause                     3.1                 da86e6ba6ca1        2 years ago         742kB

1.17.4node启动的容器和下载的镜像

#启动的容器
59b6a4cd2f44        registry.cn-hangzhou.aliyuncs.com/google_containers/coredns      "/coredns -conf /etc…"   10 hours ago        Up 10 hours                                   k8s_coredns_coredns-7f9c544f75-prdlb_kube-system_77f42046-8348-4247-b37b-6ce3dc62db18_0
c73eac71d7e4        calico/kube-controllers                                          "/usr/bin/kube-contr…"   10 hours ago        Up 10 hours                                   k8s_calico-kube-controllers_calico-kube-controllers-788d6b9876-4lls7_kube-system_25992c41-0bfe-4e84-8a04-70b63c969627_0
947909b16d67        registry.cn-hangzhou.aliyuncs.com/google_containers/coredns      "/coredns -conf /etc…"   10 hours ago        Up 10 hours                                   k8s_coredns_coredns-7f9c544f75-5p2mf_kube-system_305e69d6-6ae6-4c48-a32d-6dddc290fa8a_0
e2b467f4fb14        registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1    "/pause"                 10 hours ago        Up 10 hours                                   k8s_POD_coredns-7f9c544f75-prdlb_kube-system_77f42046-8348-4247-b37b-6ce3dc62db18_104
a5683b681459        registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1    "/pause"                 10 hours ago        Up 10 hours                                   k8s_POD_coredns-7f9c544f75-5p2mf_kube-system_305e69d6-6ae6-4c48-a32d-6dddc290fa8a_105
11a2725d9030        registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1    "/pause"                 10 hours ago        Up 10 hours                                   k8s_POD_calico-kube-controllers-788d6b9876-4lls7_kube-system_25992c41-0bfe-4e84-8a04-70b63c969627_104
a0064abf6347        calico/node                                                      "start_runit"            10 hours ago        Up 10 hours                                   k8s_calico-node_calico-node-vm4m6_kube-system_de38b605-5188-4db0-8123-061bb253be9d_0
ad86e37926e4        calico/pod2daemon-flexvol                                        "/usr/local/bin/flex…"   10 hours ago        Exited (0) 10 hours ago                       k8s_flexvol-driver_calico-node-vm4m6_kube-system_de38b605-5188-4db0-8123-061bb253be9d_0
0946ccc79ba8        6912ec2cfae6                                                     "/install-cni.sh"        10 hours ago        Exited (0) 10 hours ago                       k8s_install-cni_calico-node-vm4m6_kube-system_de38b605-5188-4db0-8123-061bb253be9d_0
fee74eaf355d        calico/cni                                                       "/opt/cni/bin/calico…"   10 hours ago        Exited (0) 10 hours ago                       k8s_upgrade-ipam_calico-node-vm4m6_kube-system_de38b605-5188-4db0-8123-061bb253be9d_0
7635ac40c020        registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1    "/pause"                 10 hours ago        Up 10 hours                                   k8s_POD_calico-node-vm4m6_kube-system_de38b605-5188-4db0-8123-061bb253be9d_0
f8dff6a9caf5        registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy   "/usr/local/bin/kube…"   10 hours ago        Up 10 hours                                   k8s_kube-proxy_kube-proxy-g8lfq_kube-system_72e60b6a-0023-4d25-b8e0-73e98bb4b1ea_0
64752d1e78b3        registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1    "/pause"                 10 hours ago        Up 10 hours                                   k8s_POD_kube-proxy-g8lfq_kube-system_72e60b6a-0023-4d25-b8e0-73e98bb4b1ea_0

#下载的镜像
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy   v1.17.4             6dec7cfde1e5        7 days ago          116MB
calico/node                                                      v3.13.1             2e5029b93d4a        7 days ago          260MB
calico/pod2daemon-flexvol                                        v3.13.1             e8c600448aae        7 days ago          111MB
calico/cni                                                       v3.13.1             6912ec2cfae6        7 days ago          207MB
calico/kube-controllers                                          v3.13.1             3971f13f2c6c        7 days ago          56.6MB
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns      1.6.5               70f311871ae1        4 months ago        41.6MB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause        3.1                 da86e6ba6ca1        2 years ago         742kB

k8s切换命名空间工具

git clone https://github.com.cnpmjs.org/ahmetb/kubectx
cp kubectx/kubens /usr/local/bin

#查看所有命名空间
kubens

#切换到kube-system命名空间
kubens kube-system
Context "kubernetes-admin@kubernetes" modified.
Active namespace is "kube-system".
泡泡吐肥皂o © gitbook.pptfz.top 2021 all right reserved,powered by Gitbook文件修订时间: 秃笔南波湾!!!

results matching ""

    No results matching ""