为什么kubernete pod报告`内存不足‘，即使主机上有空闲的内存？

问题描述

当我运行kubectl describe pod elasticsearch-master-0时，它给出以下警告：

Warning  FailedScheduling  61s (x2 over 2m30s)  default-scheduler  0/1 nodes are available: 1 Insufficient memory.

我增加了mini kube中的内存，并重新启动了mini kube，但仍然有相同的问题。

我确实运行了minikube delete，然后运行了minikube start。您可以在下面的输出中看到它使用了4个CPU和8 GB内存

 minikube v1.15.1 on Darwin 11.0.1
✨  Automatically selected the docker driver. Other choices: hyperkit, virtualbox
  Starting control plane node minikube in cluster minikube
  Creating docker container (CPUs=4, Memory=8096MB) ...
  Preparing Kubernetes v1.19.4 on Docker 19.03.13 ...
  Verifying Kubernetes components...
  Enabled addons: default-storageclass, storage-provisioner
  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

下面是从配置中获取CPU和内存的代码。

$ minikube config get cpus
4
$ minikube config get memory
8096

下面是metrics-server的输出。

$ kubectl top node
NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
minikube   466m         5%     737Mi           37%

$ kubectl top pod
W1125 20:11:23.232025   46719 top_pod.go:265] Metrics not available for pod default/elasticsearch-master-0, age: 34m3.231199s
error: Metrics not available for pod default/elasticsearch-master-0, age: 34m3.231199s

kubectl describe pod的完整输出为：

$ kubectl describe pod elasticsearch-master-0
Name:           elasticsearch-master-0
Namespace:      default
Priority:       0
Node:           <none>
Labels:         app=elasticsearch-master
                chart=elasticsearch
                controller-revision-hash=elasticsearch-master-677c65788d
                release=elasticsearch
                statefulset.kubernetes.io/pod-name=elasticsearch-master-0
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  StatefulSet/elasticsearch-master
Init Containers:
  configure-sysctl:
    Image:      docker.elastic.co/elasticsearch/elasticsearch:7.10.0
    Port:       <none>
    Host Port:  <none>
    Command:
      sysctl
      -w
      vm.max_map_count=262144
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-kthrd (ro)
Containers:
  elasticsearch:
    Image:       docker.elastic.co/elasticsearch/elasticsearch:7.10.0
    Ports:       9200/TCP, 9300/TCP
    Host Ports:  0/TCP, 0/TCP
    Limits:
      cpu:     1
      memory:  4Gi
    Requests:
      cpu:      1
      memory:   2Gi
    Readiness:  exec [sh -c #!/usr/bin/env bash -e
# If the node is starting up wait for the cluster to be ready (request params: "wait_for_status=green&timeout=1s" )
# Once it has started only check that the node itself is responding
START_FILE=/tmp/.es_start_file

# Disable nss cache to avoid filling dentry cache when calling curl
# This is required with Elasticsearch Docker using nss < 3.52
export NSS_SDB_USE_CACHE=no

http () {
  local path="${1}"
  local args="${2}"
  set -- -XGET -s

  if [ "$args" != "" ]; then
    set -- "$@" $args
  fi

  if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
    set -- "$@" -u "${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
  fi

  curl --output /dev/null -k "$@" "http://127.0.0.1:9200${path}"
}

if [ -f "${START_FILE}" ]; then
  echo 'Elasticsearch is already running, lets check the node is healthy'
  HTTP_CODE=$(http "/" "-w %{http_code}")
  RC=$?
  if [[ ${RC} -ne 0 ]]; then
    echo "curl --output /dev/null -k -XGET -s -w '%{http_code}' ${BASIC_AUTH} http://127.0.0.1:9200/ failed with RC ${RC}"
    exit ${RC}
  fi
  # ready if HTTP code 200, 503 is tolerable if ES version is 6.x
  if [[ ${HTTP_CODE} == "200" ]]; then
    exit 0
  elif [[ ${HTTP_CODE} == "503" && "7" == "6" ]]; then
    exit 0
  else
    echo "curl --output /dev/null -k -XGET -s -w '%{http_code}' ${BASIC_AUTH} http://127.0.0.1:9200/ failed with HTTP code ${HTTP_CODE}"
    exit 1
  fi

else
  echo 'Waiting for elasticsearch cluster to become ready (request params: "wait_for_status=green&timeout=1s" )'
  if http "/_cluster/health?wait_for_status=green&timeout=1s" "--fail" ; then
    touch ${START_FILE}
    exit 0
  else
    echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )'
    exit 1
  fi
fi
] delay=10s timeout=5s period=10s #success=3 #failure=3
    Environment:
      node.name:                     elasticsearch-master-0 (v1:metadata.name)
      cluster.initial_master_nodes:  elasticsearch-master-0,
      discovery.seed_hosts:          elasticsearch-master-headless
      cluster.name:                  elasticsearch
      network.host:                  0.0.0.0
      ES_JAVA_OPTS:                  -Xmx1g -Xms1g
      node.data:                     true
      node.ingest:                   true
      node.master:                   true
      node.remote_cluster_client:    true
    Mounts:
      /usr/share/elasticsearch/data from elasticsearch-master (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-kthrd (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  elasticsearch-master:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  elasticsearch-master-elasticsearch-master-0
    ReadOnly:   false
  default-token-kthrd:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-kthrd
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  76s (x2 over 77s)  default-scheduler  0/1 nodes are available: 1 Insufficient memory.

推荐答案

Minikube将在第一次启动时获取您的内存设置，但如果您以前启动时没有使用该选项，则需要执行minikube delete并重新启动。

要检查Pod/节点正在使用的资源，您可以使用minikube addons启用metrics-server：

➜  ~ minikube addons enable metrics-server 
  The 'metrics-server' addon is enabled

您必须等待一段时间才能显示指标：

➜  ~ kubectl top node 
NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
minikube   186m         4%     2344Mi          15%       
➜  ~ kubectl top pod 
NAME                     CPU(cores)   MEMORY(bytes)   
elasticsearch-master-0   6m           1272Mi