手搓跨云k3s集群 __2023年7月28日 __3条评论 __270次阅读 __0人点赞 __Tao_Qi 根据我个人跨云的经验来手搓的，有些改动。 准备操作 Debian环境准备 安装基础环境
[code]

apt install --no-install-recommends --no-install-suggests -y -qq nfs-common iptables conntrack jq socat bash-completion open-iscsi rsync ipset ipvsadm htop net-tools wget psmisc git curl nload ebtables ethtool
systemctl enable --now iscsid
mkdir -pv /etc/systemd/system.conf.d
cat > /etc/systemd/system.conf.d/30-k8s-ulimits.conf < /etc/modules-load.d/10-k8s-modules.conf <<EOF
br_netfilter
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
EOF

systemctl daemon-reload
systemctl restart systemd-modules-load
systemctl restart systemd-journald

[/code]

k3s准备
[code]

# 国外
wget https://github.com/k3s-io/k3s/releases/download/v1.27.4+k3s1/k3s
# 国内
wget https://ghproxy.com/https://github.com/k3s-io/k3s/releases/download/v1.27.4+k3s1/k3s
chmod  x k3s
mv k3s /usr/local/bin/k3s

[/code]

mysql部署
[code]

version: '2.1'

services:
  mysql:
    image: docker.io/bitnami/mysql:8.0
    container_name: mysql
    ports:
      - '3306:3306'
    volumes:
      - '/k8sdata/mysql/data:/bitnami/mysql/data'
    environment:
      # ALLOW_EMPTY_PASSWORD is recommended only for development.
      - MYSQL_ROOT_PASSWORD=cholei7quohyeaduk9OhTee1faisodaikouv8eep1iuraibuegeijiew7gue5euf
      - MYSQL_DATABASE=teil4Ir5
      - MYSQL_USER=AoQui4gion
      - MYSQL_PASSWORD=shooC9iecoh8aevaingoh2aiNeim3poorahY7too6ahru8saewaih0LizieNevie
      - MYSQL_CHARACTER_SET=utf8mb4
      - MYSQL_COLLATE=utf8mb4_general_ci
    healthcheck:
      test: ['CMD', '/opt/bitnami/scripts/mysql/healthcheck.sh']
      interval: 15s
      timeout: 5s
      retries: 6

volumes:
  mysql_data:
    driver: local

[/code]

启动mysql
[code]

docker-compose up -d

[/code]

k3s 部署

• 虽然新版本k3s内置tailscale支持，但是我发现基于跨云tailscale组网，不支持headscale组网，且导致哪吒监控数据流量比较大，暂时就不考虑了。
• 容器的runtime最好都一致，要不都用docker，要不都是默认内置的containerd
• 数据存储可以选择熟悉的数据库，我这里选择mysql，master高可用

master节点配置 给个示例的 /etc/systemd/system/k3s.service
[code]

[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target

[Install]
WantedBy=multi-user.target

[Service]
Type=notify
EnvironmentFile=-/etc/systemd/system/k3s.service.env
KillMode=process
Delegate=yes
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s server \
                --tls-san t1.slb.ns.local \
                --tls-san t2.slb.ns.local \
                --tls-san t1.clb.ns.local \
                --tls-san t2.clb.ns.local \
                --datastore-endpoint mysql://root:cholei7quohyeaduk9OhTee1faisodaikouv8eep1iuraibuegeijiew7gue5euf@tcp(db.slb.ns.local:3306)/ \
                --node-external-ip  \
                --https-listen-port 26443 \
                --cluster-cidr 10.142.0.0/16 \
                --service-cidr 10.143.0.0/16 \
                --flannel-external-ip \
                --flannel-backend wireguard-native \
                --server https://t1.slb.ns.local:26443 \
                --token we3ohCh1ahjeng6goo9hahke0Agoox5a \
                --docker \
                --multi-cluster-cidr \
                --disable-network-policy \
                --disable-helm-controller \
                --disable servicelb,traefik \
                --kube-proxy-arg "proxy-mode=ipvs" "masquerade-all=true" \
                --kube-proxy-arg "metrics-bind-address=0.0.0.0"

[/code]

worker 节点配置
[code]

[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target

[Install]
WantedBy=multi-user.target

[Service]
Type=exec
EnvironmentFile=-/etc/systemd/system/k3s.service.env
KillMode=process
Delegate=yes
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s agent \
        --node-external-ip  \
        --server https://t1.clb.ns.local:26443 \
        --token we3ohCh1ahjeng6goo9hahke0Agoox5a \
        --docker \
        --kube-proxy-arg "proxy-mode=ipvs" "masquerade-all=true" \
        --kube-proxy-arg "metrics-bind-address=0.0.0.0"

[/code]

开机并启动
[code]

systemctl enable k3s --now

[/code]

其他组件安装

• 跨云分布式存储
• 内网负载均衡

如果对这些有想法可以再写过文档 最后 不想手搓的可以用rancher或者 https://github.com/cnrancher/autok3s 这两个都是有UI的，我推荐用后者 本作品采用 知识共享署名-相同方式共享 4.0 国际许可协议

1. sduoduo说道： 2023年7月28日 下午5:34
2. songshi说道： 2023年7月28日 下午5:35
3. ananas说道： 2023年7月28日 下午5:38

发表回复 取消回复电子邮件地址不会被公开。必填项已用 标注 __ __