Kubernetes Pod中的i/o超时问题是什么?

我部署了一个带有两个EC2实例的Kubernetes集群。

$ kubectl get nodes
NAME              STATUS   ROLES                  AGE   VERSION
ip-172-31-21-12   Ready    control-plane,master   36h   v1.20.2
ip-172-31-21-62   Ready    <none>                 12h   v1.20.2

集群中有一个错误pod:

$ kubectl get pods
NAME                      READY   STATUS    RESTARTS   AGE
es-cluster-0              0/1     Error     141        12h

我在这个pod的日志中看到了这个错误:

$ kubectl logs pods/es-cluster-0
Error from server: Get "https://172.31.21.62:10250/containerLogs/default/es-cluster-0/elasticsearch": dial tcp 172.31.21.62:10250: i/o timeout

它指的是节点出现错误 172.31.21.62:10250 我该如何解决这个问题?我不确定这个错误是否意味着?

以下命令来自主节点:

kubectl describe pods/es-cluster-0 命令输出:

$ kubectl describe pods/es-cluster-0
Name:         es-cluster-0
Namespace:    default
Priority:     0
Node:         ip-172-31-21-62/172.31.21.62
Start Time:   Wed, 17 Feb 2021 10:08:04 +0000
Labels:       controller-revision-hash=es-cluster-55b6944c56
              name=elasticsearch
              statefulset.kubernetes.io/pod-name=es-cluster-0
Annotations:  <none>
Status:       Running
IP:           10.32.0.2
IPs:
  IP:           10.32.0.2
Controlled By:  StatefulSet/es-cluster
Containers:
  elasticsearch:
    Container ID:   docker://838e02ff6fba31234656e68b804f49e86ec7fea0053e5d1062abdd9d24b728b9
    Image:          elasticsearch:7.10.1
    Image ID:       docker-pullable://elasticsearch@sha256:7cd88158f6ac75d43b447fdd98c4eb69483fa7bf1be5616a85fe556262dc864a
    Ports:          9200/TCP, 9300/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    78
      Started:      Thu, 18 Feb 2021 01:40:23 +0000
      Finished:     Thu, 18 Feb 2021 01:40:39 +0000
    Ready:          False
    Restart Count:  178
    Environment:    <none>
    Mounts:
      /usr/share/elasticsearch/config/elasticsearch.yml from elasticsearch-config (rw,path="elasticsearch.yml")
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ntgwr (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  elasticsearch-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      es-config
    Optional:  false
  default-token-ntgwr:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-ntgwr
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason   Age                   From     Message
  ----     ------   ----                  ----     -------
  Warning  BackOff  42s (x4082 over 15h)  kubelet  Back-off restarting failed container

以下命令来自主节点:

$ netstat -tunlp| grep 10250
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6       0      0 :::10250                :::*                    LISTEN      -

以下命令来自worker节点:

$ kubectl describe pods/es-cluster-0
Name:         es-cluster-0
Namespace:    default
Priority:     0
Node:         ip-172-31-21-62/172.31.21.62
Start Time:   Wed, 17 Feb 2021 10:08:04 +0000
Labels:       controller-revision-hash=es-cluster-55b6944c56
              name=elasticsearch
              statefulset.kubernetes.io/pod-name=es-cluster-0
Annotations:  <none>
Status:       Running
IP:           10.32.0.2
IPs:
  IP:           10.32.0.2
Controlled By:  StatefulSet/es-cluster
Containers:
  elasticsearch:
    Container ID:   docker://0efa71da7b70725b248cfe8dfae5560c2fc95aaf40e3a3a899970e8579e7ac27
    Image:          elasticsearch:7.10.1
    Image ID:       docker-pullable://elasticsearch@sha256:7cd88158f6ac75d43b447fdd98c4eb69483fa7bf1be5616a85fe556262dc864a
    Ports:          9200/TCP, 9300/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    78
      Started:      Thu, 18 Feb 2021 05:31:55 +0000
      Finished:     Thu, 18 Feb 2021 05:32:12 +0000
    Ready:          False
    Restart Count:  221
    Environment:    <none>
    Mounts:
      /usr/share/elasticsearch/config/elasticsearch.yml from elasticsearch-config (rw,path="elasticsearch.yml")
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ntgwr (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  elasticsearch-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      es-config
    Optional:  false
  default-token-ntgwr:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-ntgwr
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason   Age                   From     Message
  ----     ------   ----                  ----     -------
  Normal   Pulled   55m (x211 over 19h)   kubelet  Container image "elasticsearch:7.10.1" already present on machine
  Warning  BackOff  54s (x5087 over 19h)  kubelet  Back-off restarting failed container

以下命令来自worker节点:

$ netstat -tunlp| grep 10250
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6       0      0 :::10250                :::*                    LISTEN      -