用Hashicorp Vault搭建自己的CA（六）——用Kubernetes验证身份（上）

2024-10-24 约 3000 字预计阅读 6 分钟

https://cdn.miyunda.net/uPic/vault-ca-in-kubernetes-banner_20241031.png

上一篇文章中我们用AppRole方式成功滴使Kubernetes集群在Vault中验证身份，配合安全策略获得了Vault中的CA签发的数字证书。这种方法有个缺点，就是安全令牌（token）是有寿命的，过家家的时候我们可以设置为永不过期，但是生产环境里肯定不能这么搞。这玩意过期了就得有人生成新的，再传递给Kubernetes管理员，去更新信息。

Vault支持的验证方式有很多，Kubernetes验证就是其中之一。

Vault in Kubernetes

C 实验环境

先列出动手实验的计算机信息:

我在一台运行Debian 12的虚拟机安装的Minikube：它采用默认方式启动容器，也就是Docker。

ℹ️ 再次强烈推荐macOS用户用OrbStack来代替Minikube，又好用又轻👍。

Minikube版本：

1
2


minikube version: v1.33.1
commit: 5883c09216182566a63dff4c326a6fc9ed2982ff

清理（可选）：

1
2
3


minikube stop
minikube delete
minikube start

Docker版本：

1
2
3
4
5
6


docker version
Client: Docker Engine - Community
 Version:           27.1.1
 API version:       1.46
 Go version:        go1.21.12
...

Helm是用apt 工具安装的：

1
2
3
4
5
6


curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
sudo apt-get install apt-transport-https --yes
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm
helm version

1

helm versionversion.BuildInfo{Version:"v3.15.3", GitCommit:"3bb50bbbdd9c946ba9989fbe4fb4104766302a64", GitTreeState:"clean", GoVersion:"go1.22.5"}

引入Hashicorp官方源：

1
2


helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

验证下：

1
2
3
4


helm search repo hashicorp/vault
NAME                            	CHART VERSION	APP VERSION	DESCRIPTION
hashicorp/vault                 	0.28.1       	1.17.2     	Official HashiCorp Vault Chart
hashicorp/vault-secrets-operator	0.9.0        	0.9.0      	Official Vault Secrets Operator Chart

安装Vault

在Minikube机器上准备一个文件夹：

1

mkdir ~/vault-in-minikube && cd ~/vault-in-minikube

新安装一个Vault：

1

helm install vault hashicorp/vault --set "injector.enabled=false"

看看状态，可以看到还没好：

1
2
3


kubectl get pods
NAME      READY   STATUS              RESTARTS   AGE
vault-0   0/1     ContainerCreating   0          14s

初始化一下，走个简易的手续就行，安全凭据存在文件里：

1
2
3
4


kubectl exec vault-0 -- vault operator init \
    -key-shares=1 \
    -key-threshold=1 \
    -format=json > vault-in-k8s-key.json

⚠️ 只生成单个解封key是不靠谱的，不要在生产环境里面这么搞，过家家无所谓。

读取解封的key：

1

VAULT_UNSEAL_KEY=$(jq -r ".unseal_keys_b64[]" vault-in-k8s-key.json)

注：没有jq的自己装一下。

解封：

1

kubectl exec vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY

这回应该好了：

1
2
3


kubectl get pods
NAME      READY   STATUS    RESTARTS   AGE
vault-0   1/1     Running   0          20m

读取新装Vault管理员（root）的token：

1

VAULT_ROOT_TOKEN=$(jq -r ".root_token" vault-in-k8s-key.json)

登录咧：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


kubectl exec vault-0 -- vault login $VAULT_ROOT_TOKEN
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvs.tLsjfeZiePgde8f5fwAVsCtO
token_accessor       OESG7tyqJ9jxm4gQp9TK3vKd
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

建立中间（intermediate）CA：

我们将在此Vault中建立中间CA，并由一个外部的根（root）CA来为它签发中间CA的证书。

建立CA：

1
2


kubectl exec vault-0 -- vault secrets enable -path=int-in-k8s pki
Success! Enabled the pki secrets engine at: int-in-k8s/

随便微调下：

1
2


kubectl exec vault-0 -- vault secrets tune -max-lease-ttl=8760h int-in-k8s
Success! Tuned the secrets engine at: int-in-k8s/

生成CSR：

1
2
3
4


kubectl exec  vault-0 -- vault write -format=json int-in-k8s/intermediate/generate/internal \
     common_name="Vault in k8s Intermediate Authority" \
     issuer_name="int-in-k8s" \
     | jq -r '.data.csr' > int-in-k8s.csr

将这个CSR文件复制到根CA那里，我就用之前几篇文章的那个Vault里的根CA。那么就把CSR文件复制到能访问老Vault的计算机里。

1
2


export VAULT_ADDR='https://vault-dev.miyunda.com' # 引号里面替换成自己的FQDN/IP地址和端口号
vault login

签发中间证书：

1
2
3
4
5


vault write -format=json <secret engine path>/root/sign-intermediate \
     issuer_ref="<issuer_name>" \
     csr=@int-in-k8s.csr \
     format=pem_bundle ttl="87600h" \
     | jq -r '.data.certificate' > int-in-k8s.cert.pem

假如忘记了issuer_name：

1
2


vault read <secret engine path>/issuer/$(vault list -format=json <secret engine path>/issuers/ | jq -r '.[]') \
 | tail -n 11

假如连Secret Engine的路径也忘了:

1

vault secrets list

好奇的话可以看看签发的中间CA证书：

1

openssl x509 -in int-in-k8s.cert.pem -text

把它复制回Minikube计算机，用什么方法都行，前面的CSR和这个证书都不含敏感信息。

然后复制到容器里并导入：

1

kubectl cp int-in-k8s.cert.pem vault-0:/tmp

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


kubectl exec  vault-0 -- vault write int-in-k8s/intermediate/set-signed certificate=@/tmp/int-in-k8s.cert.pem
Key                 Value
---                 -----
existing_issuers    <nil>
existing_keys       <nil>
imported_issuers    [79ab0305-cd2d-cfa7-be78-dcde40a4eeb0 a463f347-7710-aeba-6ee3-cd0ececfb631]
imported_keys       <nil>
mapping             map[79ab0305-cd2d-cfa7-be78-dcde40a4eeb0:a6a18e58-859b-d058-a770-b0681974852b a463f347-7710-aeba-6ee3-cd0ececfb631:]
WARNING! The following warnings were returned from Vault:

  * This mount hasn't configured any authority information access (AIA)
  fields; this may make it harder for systems to find missing certificates
  in the chain or to validate revocation status of certificates. Consider
  updating /config/urls or the newly generated issuer with this information.

洁癖患者也许不喜欢容器里面有多余的东西：

1

kubectl exec vault-0 -- rm /tmp/int-in-k8s.cert.pem

把URL设置下：

1
2
3


kubectl exec vault-0 -- vault write int-in-k8s/config/urls \
    issuing_certificates="http://vault.default:8200/v1/int-in-k8s/ca" \
    crl_distribution_points="http://vault.default:8200/v1/int-in-k8s/crl"

在中间CA下面新建一个角色（role）：

1
2
3


kubectl exec vault-0 -- vault write int-in-k8s/roles/miyunda-com \
    allowed_domains=miyunda.com \
    allow_subdomains=true max_ttl=72h

还得解决权限问题：

1

kubectl exec --stdin=true --tty=true vault-0 -- /bin/sh

1
2
3
4
5


vault policy write int-in-k8s-acl - <<EOF
path "int-in-k8s*"                        { capabilities = ["read", "list"] }
path "int-in-k8s/sign/miyunda-com"    { capabilities = ["create", "update"] }
path "int-in-k8s/issue/miyunda-com"   { capabilities = ["create"] }
EOF

不想给老CA作中间CA的可以建个新的根CA，后面到Kubernetes上记得自己把相应的路径都改下

1

kubectl exec --stdin=true --tty=true vault-0 -- /bin/sh

1
2
3


vault write ca-root-k8s/root/generate/internal \
    common_name=miyunda.com \
    ttl=8760h

1
2
3


vault write ca-root-k8s/config/urls \
    issuing_certificates="http://vault.default:8200/v1/ca-root-k8s/ca" \
    crl_distribution_points="http://vault.default:8200/v1/ca-root-k8s/crl"

1
2
3
4


vault write ca-root-k8s/roles/miyunda-com \
    allowed_domains=miyunda.com \
    allow_subdomains=true \
    max_ttl=72h

1
2
3
4
5


vault policy write ca-root-k8s-acl - <<EOF
path "ca-root-k8s*"                        { capabilities = ["read", "list"] }
path "ca-root-k8s/sign/miyunda-com"    { capabilities = ["create", "update"] }
path "ca-root-k8s/issue/miyunda-com"   { capabilities = ["create"] }
EOF

配置Kubernetes验证方式

不要退出容器的命令行，继续搞，开Kubernetes验证方式：

1

vault auth enable kubernetes

下一步是告诉Vault去哪找Kubernetes集群的API，因为Vault就运行在Kubernetes集群内部，所以可以轻松地与集群API交流。

1
2
3


vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"
Success! Data written to: auth/kubernetes/config

下一步是在此验证方式内创建一个“角色”（可以理解为一个给机器用的账号）并赋权。

1
2
3
4
5


vault write auth/kubernetes/role/int-in-k8s-issuer \
    bound_service_account_names=int-in-k8s-issuer \
    bound_service_account_namespaces=default \
    policies=int-in-k8s-acl \
    ttl=24h

服务账号

退出容器的命令行，在Kubernetes这边建立服务账号：

1

kubectl create serviceaccount int-in-k8s-issuer

1
2
3
4
5
6
7
8
9


cat >> int-in-k8s-issuer-secret.yaml <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: issuer-token-abcde
  annotations:
    kubernetes.io/service-account.name: int-in-k8s-issuer
type: kubernetes.io/service-account-token
EOF

下面的secret是给Kubernetes的服务账号用的，里面有个token，Vault会用这个token去访问Kubernetes API。

1

kubectl apply -f int-in-k8s-issuer-secret.yaml

1
2
3
4


kubectl get secrets
NAME                          TYPE                                  DATA   AGE
issuer-token-abcde            kubernetes.io/service-account-token   3      13s
sh.helm.release.v1.vault.v1   helm.sh/release.v1                    1      761      17h

证书签发

（退出容器的命令行）安装cert-manager，由于上一篇文章我们有过经验，就不解释了：

1
2
3
4
5
6
7
8


helm repo add jetstack https://charts.jetstack.io --force-update
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.15.3 \
  --set crds.enabled=true \
  --set prometheus.enabled=false

1

kubectl create namespace cert-manager

1
2
3
4
5


kubectl get pods --namespace cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-69d959bd45-vpgg8              1/1     Running   0          4m41s
cert-manager-cainjector-5d8798687c-6sh54   1/1     Running   0          4m41s
cert-manager-webhook-c77744d75-q47q4       1/1     Running   0          4m41s

1

ISSUER_SECRET_REF=$(kubectl get secrets --output=json | jq -r '.items[].metadata | select(.name|startswith("issuer-token-a")).name')

和之前几篇文章一样——得让cert-manager知道去哪里申请证书：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


cat > int-in-k8s-issuer.yaml <<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: int-in-k8s-issuer
  namespace: default
spec:
  vault:
    server: http://vault.default:8200
    path: int-in-k8s/sign/miyunda-com
    auth:
      kubernetes:
        mountPath: /v1/auth/kubernetes
        role: int-in-k8s-issuer
        secretRef:
          name: $ISSUER_SECRET_REF
          key: token
EOF

1

kubectl apply -f int-in-k8s-issuer.yaml

1
2
3


kubectl get issuers
NAME                READY   AGE
int-in-k8s-issuer   True    4s

尝试手搓一个证书请求：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


cat > hellok8s-miyunda-com.yaml <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: hellok8s-miyunda-com
  namespace: default
spec:
  secretName: hellok8s-miyunda-com-tls
  issuerRef:
    name: int-in-k8s-issuer
  commonName: hellok8s.miyunda.com
  dnsNames:
  - hellok8s.miyunda.com
EOF

1

kubectl apply --filename hellok8s-miyunda-com.yaml

可以看到新证书已经就绪了：

1
2
3


kubectl get certificate
NAME                   READY   SECRET                     AGE
hellok8s-miyunda-com   True    hellok8s-miyunda-com-tls   6s

看看它的详细信息：

1

kubectl describe certificate hellok8s-miyunda-com

也可以让ingress通过cert-manger获取证书，与上一篇文章类似，不多解释了：

1

minikube addons enable ingress

1

nano ingress.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kuard
  annotations:
    cert-manager.io/issuer: "int-in-k8s-issuer"
    cert-manager.io/common-name: "hellonerd.miyunda.com"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - hellonerd.miyunda.com
    secretName: hellonerd-int-in-k8s-tls
  rules:
  - host: hellonerd.miyunda.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kuard
            port:
              number: 80

1

kubectl apply -f ingress.yaml

1
2
3
4
5


kubectl get certificate

NAME                       READY   SECRET                     AGE
hellok8s-miyunda-com       True    hellok8s-miyunda-com-tls   5m43s
hellonerd-int-in-k8s-tls   True    hellonerd-int-in-k8s-tls   9s

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50


kubectl describe certificate hellonerd-int-in-k8s-tls
Name:         hellonerd-int-in-k8s-tls
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2024-10-20T14:39:14Z
  Generation:          1
  Owner References:
    API Version:           networking.k8s.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  kuard
    UID:                   31432297-7621-4b16-bcec-0c81044bb3d5
  Resource Version:        5562
  UID:                     b4eb5088-d60f-4739-9393-e64d7e1f1856
Spec:
  Common Name:  hellonerd.miyunda.com
  Dns Names:
    hellonerd.miyunda.com
  Issuer Ref:
    Group:      cert-manager.io
    Kind:       Issuer
    Name:       int-in-k8s-issuer
  Secret Name:  hellonerd-int-in-k8s-tls
  Usages:
    digital signature
    key encipherment
Status:
  Conditions:
    Last Transition Time:  2024-10-20T14:39:14Z
    Message:               Certificate is up to date and has not expired
    Observed Generation:   1
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2024-10-23T14:39:14Z
  Not Before:              2024-10-20T14:38:44Z
  Renewal Time:            2024-10-22T14:39:04Z
  Revision:                1
Events:
  Type    Reason     Age   From                                       Message
  ----    ------     ----  ----                                       -------
  Normal  Issuing    74s   cert-manager-certificates-trigger          Issuing certificate as Secret does not exist
  Normal  Generated  74s   cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "hellonerd-int-in-k8s-tls-ljkzh"
  Normal  Requested  74s   cert-manager-certificates-request-manager  Created new CertificateRequest resource "hellonerd-int-in-k8s-tls-1"
  Normal  Issuing    74s   cert-manager-certificates-issuing          The certificate has been successfully issued

尝试去访问ingress看看，里面的IP地址换成$(minikube ip)也可以：

1
2
3
4
5


curl -o my-ca.crt https://vault-dev.miyunda.com/v1/ca-root-x1/ca/pem
curl -ivL \
    --cacert my-ca.crt \
    --resolve hellonerd.miyunda.com:443:192.168.49.2 \
    https://hellonerd.miyunda.com

返回的结果符合预期：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=hellonerd.miyunda.com
*  start date: Oct 20 14:38:44 2024 GMT
*  expire date: Oct 23 14:39:14 2024 GMT
*  subjectAltName: host "hellonerd.miyunda.com" matched cert's "hellonerd.miyunda.com"
*  issuer: CN=Vault in k8s Intermediate Authority
*  SSL certificate verify ok.
* using HTTP/2

以上记录了我在Kubernetes内部运行Vault并签发证书的过程，它不用手动更新安全凭据，减少了管理员的工作量，提高了安全性；但是这种方法的缺点也是明显的：它需要每个Kubernetes集群都安装一个Vault，集群多了就很烦。下一篇文章我们仍然让Vault使用相同的方式验证，但将Vault移出Kubernetes集群。

好了，看过了就等于会了。感谢观看！😽