Getting started

• 1: Release notes and version skew
• 1.1: v1.21 Release Notes
• 1.2: Kubernetes version and version skew support policy
• 2: Learning environment
• 3: Production environment
• 3.1: Container runtimes
• 3.2: Installing Kubernetes with deployment tools
• 3.2.1: Bootstrapping clusters with kubeadm
• 3.2.1.1: Installing kubeadm
• 3.2.1.2: Troubleshooting kubeadm
• 3.2.1.3: Creating a cluster with kubeadm
• 3.2.1.4: Customizing control plane configuration with kubeadm
• 3.2.1.5: Options for Highly Available topology
• 3.2.1.6: Creating Highly Available clusters with kubeadm
• 3.2.1.7: Set up a High Availability etcd cluster with kubeadm
• 3.2.1.8: Configuring each kubelet in your cluster using kubeadm
• 3.2.1.9: Dual-stack support with kubeadm
• 3.2.2: Installing Kubernetes with kops
• 3.2.3: Installing Kubernetes with Kubespray
• 3.3: Turnkey Cloud Solutions
• 3.4: Windows in Kubernetes
• 3.4.1: Intro to Windows support in Kubernetes
• 3.4.2: Guide for scheduling Windows containers in Kubernetes
• 4: Best practices
• 4.1: Considerations for large clusters
• 4.2: Running in multiple zones
• 4.3: Validate node setup
• 4.4: PKI certificates and requirements

1 - Release notes and version skew

1.1 - v1.21 Release Notes

v1.21.0

Documentation

Downloads for v1.21.0

Source Code

Client Binaries

Server Binaries

Node Binaries

Changelog since v1.20.0

What's New (Major Themes)

Deprecation of PodSecurityPolicy

PSP as an admission controller resource is being deprecated. Deployed PodSecurityPolicy's will keep working until version 1.25, their target removal from the codebase. A new feature, with a working title of "PSP replacement policy", is being developed in KEP-2579. To learn more, read PodSecurityPolicy Deprecation: Past, Present, and Future.

Kubernetes API Reference Documentation

The API reference is now generated with gen-resourcesdocs and it is moving to Kubernetes API

Kustomize Updates in Kubectl

Kustomize version in kubectl had a jump from v2.0.3 to v4.0.5. Kustomize is now treated as a library and future updates will be less sporadic.

Default Container Labels

Pod with multiple containers can use kubectl.kubernetes.io/default-container label to have a container preselected for kubectl commands. More can be read in KEP-2227.

Immutable Secrets and ConfigMaps

Immutable Secrets and ConfigMaps graduates to GA. This feature allows users to specify that the contents of a particular Secret or ConfigMap is immutable for its object lifetime. For such instances, Kubelet will not watch/poll for changes and therefore reducing apiserver load.

Structured Logging in Kubelet

Kubelet has adopted structured logging, thanks to community effort in accomplishing this within the release timeline. Structured logging in the project remains an ongoing effort -- for folks interested in participating, keep an eye / chime in to the mailing list discussion.

Storage Capacity Tracking

Traditionally, the Kubernetes scheduler was based on the assumptions that additional persistent storage is available everywhere in the cluster and has infinite capacity. Topology constraints addressed the first point, but up to now pod scheduling was still done without considering that the remaining storage capacity may not be enough to start a new pod. Storage capacity tracking addresses that by adding an API for a CSI driver to report storage capacity and uses that information in the Kubernetes scheduler when choosing a node for a pod. This feature serves as a stepping stone for supporting dynamic provisioning for local volumes and other volume types that are more capacity constrained.

Generic Ephemeral Volumes

Generic ephermeral volumes feature allows any existing storage driver that supports dynamic provisioning to be used as an ephemeral volume with the volume’s lifecycle bound to the Pod. It can be used to provide scratch storage that is different from the root disk, for example persistent memory, or a separate local disk on that node. All StorageClass parameters for volume provisioning are supported. All features supported with PersistentVolumeClaims are supported, such as storage capacity tracking, snapshots and restore, and volume resizing.

CSI Service Account Token

CSI Service Account Token feature moves to Beta in 1.21. This feature improves the security posture and allows CSI drivers to receive pods' bound service account tokens. This feature also provides a knob to re-publish volumes so that short-lived volumes can be refreshed.

CSI Health Monitoring

The CSI health monitoring feature is being released as a second Alpha in Kubernetes 1.21. This feature enables CSI Drivers to share abnormal volume conditions from the underlying storage systems with Kubernetes so that they can be reported as events on PVCs or Pods. This feature serves as a stepping stone towards programmatic detection and resolution of individual volume health issues by Kubernetes.

Known Issues

TopologyAwareHints feature falls back to default behavior

The feature gate currently falls back to the default behavior in most cases. Enabling the feature gate will add hints to EndpointSlices, but functional differences are only observed in non-dual stack kube-proxy implementation. The fix will be available in coming releases.

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Kube-proxy's IPVS proxy mode no longer sets the net.ipv4.conf.all.route_localnet sysctl parameter. Nodes upgrading will have net.ipv4.conf.all.route_localnet set to 1 but new nodes will inherit the system default (usually 0). If you relied on any behavior requiring net.ipv4.conf.all.route_localnet, you must set ensure it is enabled as kube-proxy will no longer set it automatically. This change helps to further mitigate CVE-2020-8558. (#92938, @lbernail) [SIG Network and Release]
• Kubeadm: during "init" an empty cgroupDriver value in the KubeletConfiguration is now always set to "systemd" unless the user is explicit about it. This requires existing machine setups to configure the container runtime to use the "systemd" driver. Documentation on this topic can be found here: https://kubernetes.io/docs/setup/production-environment/container-runtimes/. When upgrading existing clusters / nodes using "kubeadm upgrade" the old cgroupDriver value is preserved, but in 1.22 this change will also apply to "upgrade". For more information on migrating to the "systemd" driver or remaining on the "cgroupfs" driver see: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/. (#99471, @neolit123) [SIG Cluster Lifecycle]
• Newly provisioned PVs by EBS plugin will no longer use the deprecated "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" labels. It will use "topology.kubernetes.io/zone" and "topology.kubernetes.io/region" labels instead. (#99130, @ayberk) [SIG Cloud Provider, Storage and Testing]
• Newly provisioned PVs by OpenStack Cinder plugin will no longer use the deprecated "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" labels. It will use "topology.kubernetes.io/zone" and "topology.kubernetes.io/region" labels instead. (#99719, @jsafrane) [SIG Cloud Provider and Storage]
• Newly provisioned PVs by gce-pd will no longer have the beta FailureDomain label. gce-pd volume plugin will start to have GA topology label instead. (#98700, @Jiawei0227) [SIG Cloud Provider, Storage and Testing]
• OpenStack Cinder CSI migration is on by default, Clinder CSI driver must be installed on clusters on OpenStack for Cinder volumes to work. (#98538, @dims) [SIG Storage]
• Remove alpha CSIMigrationXXComplete flag and add alpha InTreePluginXXUnregister flag. Deprecate CSIMigrationvSphereComplete flag and it will be removed in v1.22. (#98243, @Jiawei0227)
• Remove storage metrics storage_operation_errors_total, since we already have storage_operation_status_count.And add new field status for storage_operation_duration_seconds, so that we can know about all status storage operation latency. (#98332, @JornShen) [SIG Instrumentation and Storage]
• The metric storage_operation_errors_total is not removed, but is marked deprecated, and the metric storage_operation_status_count is marked deprecated. In both cases the storage_operation_duration_seconds metric can be used to recover equivalent counts (using status=fail-unknown in the case of storage_operations_errors_total). (#99045, @mattcary)
• ServiceNodeExclusion, NodeDisruptionExclusion and LegacyNodeRoleBehavior features have been promoted to GA. ServiceNodeExclusion and NodeDisruptionExclusion are now unconditionally enabled, while LegacyNodeRoleBehavior is unconditionally disabled. To prevent control plane nodes from being added to load balancers automatically, upgrade users need to add "node.kubernetes.io/exclude-from-external-load-balancers" label to control plane nodes. (#97543, @pacoxu)

Changes by Kind

Deprecation

• Aborting the drain command in a list of nodes will be deprecated. The new behavior will make the drain command go through all nodes even if one or more nodes failed during the drain. For now, users can try such experience by enabling --ignore-errors flag. (#98203, @yuzhiquan)

• Delete deprecated service.beta.kubernetes.io/azure-load-balancer-mixed-protocols mixed procotol annotation in favor of the MixedProtocolLBService feature (#97096, @nilo19) [SIG Cloud Provider]

• Deprecate the topologyKeys field in Service. This capability will be replaced with upcoming work around Topology Aware Subsetting and Service Internal Traffic Policy. (#96736, @andrewsykim) [SIG Apps]

• Kube-proxy: remove deprecated --cleanup-ipvs flag of kube-proxy, and make --cleanup flag always to flush IPVS (#97336, @maaoBit) [SIG Network]

• Kubeadm: deprecated command "alpha selfhosting pivot" is now removed. (#97627, @knight42)

• Kubeadm: graduate the command kubeadm alpha kubeconfig user to kubeadm kubeconfig user. The kubeadm alpha kubeconfig user command is deprecated now. (#97583, @knight42) [SIG Cluster Lifecycle]

• Kubeadm: the "kubeadm alpha certs" command is removed now, please use "kubeadm certs" instead. (#97706, @knight42) [SIG Cluster Lifecycle]

• Kubeadm: the deprecated kube-dns is no longer supported as an option. If "ClusterConfiguration.dns.type" is set to "kube-dns" kubeadm will now throw an error. (#99646, @rajansandeep) [SIG Cluster Lifecycle]

• Kubectl: The deprecated kubectl alpha debug command is removed. Use kubectl debug instead. (#98111, @pandaamanda) [SIG CLI]

• Official support to build kubernetes with docker-machine / remote docker is removed. This change does not affect building kubernetes with docker locally. (#97935, @adeniyistephen) [SIG Release and Testing]

• Remove deprecated --generator, --replicas, --service-generator, --service-overrides, --schedule from kubectl run Deprecate --serviceaccount, --hostport, --requests, --limits in kubectl run (#99732, @soltysh)

• Remove the deprecated metrics "scheduling_algorithm_preemption_evaluation_seconds" and "binding_duration_seconds", suggest to use "scheduler_framework_extension_point_duration_seconds" instead. (#96447, @chendave) [SIG Cluster Lifecycle, Instrumentation, Scheduling and Testing]

• Removing experimental windows container hyper-v support with Docker (#97141, @wawa0210) [SIG Node and Windows]

• Rename metrics etcd_object_counts to apiserver_storage_object_counts and mark it as stable. The original etcd_object_counts metrics name is marked as "Deprecated" and will be removed in the future. (#99785, @erain) [SIG API Machinery, Instrumentation and Testing]

• The GA TokenRequest and TokenRequestProjection feature gates have been removed and are unconditionally enabled. Remove explicit use of those feature gates in CLI invocations. (#97148, @wawa0210) [SIG Node]

• The PodSecurityPolicy API is deprecated in 1.21, and will no longer be served starting in 1.25. (#97171, @deads2k) [SIG Auth and CLI]

• The batch/v2alpha1 CronJob type definitions and clients are deprecated and removed. (#96987, @soltysh) [SIG API Machinery, Apps, CLI and Testing]

• The export query parameter (inconsistently supported by API resources and deprecated in v1.14) is fully removed. Requests setting this query parameter will now receive a 400 status response. (#98312, @deads2k) [SIG API Machinery, Auth and Testing]

• audit.k8s.io/v1beta1 and audit.k8s.io/v1alpha1 audit policy configuration and audit events are deprecated in favor of audit.k8s.io/v1, available since v1.13. kube-apiserver invocations that specify alpha or beta policy configurations with --audit-policy-file, or explicitly request alpha or beta audit events with --audit-log-version / --audit-webhook-version must update to use audit.k8s.io/v1 and accept audit.k8s.io/v1 events prior to v1.24. (#98858, @carlory) [SIG Auth]

• discovery.k8s.io/v1beta1 EndpointSlices are deprecated in favor of discovery.k8s.io/v1, and will no longer be served in Kubernetes v1.25. (#100472, @liggitt)

• diskformat storage class parameter for in-tree vSphere volume plugin is deprecated as of v1.21 release. Please consider updating storageclass and remove diskformat parameter. vSphere CSI Driver does not support diskformat storageclass parameter.

  vSphere releases less than 67u3 are deprecated as of v1.21. Please consider upgrading vSphere to 67u3 or above. vSphere CSI Driver requires minimum vSphere 67u3.

  VM Hardware version less than 15 is deprecated as of v1.21. Please consider upgrading the Node VM Hardware version to 15 or above. vSphere CSI Driver recommends Node VM's Hardware version set to at least vmx-15.

  Multi vCenter support is deprecated as of v1.21. If you have a Kubernetes cluster spanning across multiple vCenter servers, please consider moving all k8s nodes to a single vCenter Server. vSphere CSI Driver does not support Kubernetes deployment spanning across multiple vCenter servers.

  Support for these deprecations will be available till Kubernetes v1.24. (#98546, @divyenpatel)

API Change

• 1. PodAffinityTerm includes a namespaceSelector field to allow selecting eligible namespaces based on their labels.
  2. A new CrossNamespacePodAffinity quota scope API that allows restricting which namespaces allowed to use PodAffinityTerm with corss-namespace reference via namespaceSelector or namespaces fields. (#98582, @ahg-g) [SIG API Machinery, Apps, Auth and Testing]
• Add Probe-level terminationGracePeriodSeconds field (#99375, @ehashman) [SIG API Machinery, Apps, Node and Testing]
• Added .spec.completionMode field to Job, with accepted values NonIndexed (default) and Indexed. This is an alpha field and is only honored by servers with the IndexedJob feature gate enabled. (#98441, @alculquicondor) [SIG Apps and CLI]
• Adds support for endPort field in NetworkPolicy (#97058, @rikatz) [SIG Apps and Network]
• CSIServiceAccountToken graduates to Beta and enabled by default. (#99298, @zshihang)
• Cluster admins can now turn off /debug/pprof and /debug/flags/v endpoint in kubelet by setting enableProfilingHandler and enableDebugFlagsHandler to false in the Kubelet configuration file. Options enableProfilingHandler and enableDebugFlagsHandler can be set to true only when enableDebuggingHandlers is also set to true. (#98458, @SaranBalaji90)
• DaemonSets accept a MaxSurge integer or percent on their rolling update strategy that will launch the updated pod on nodes and wait for those pods to go ready before marking the old out-of-date pods as deleted. This allows workloads to avoid downtime during upgrades when deployed using DaemonSets. This feature is alpha and is behind the DaemonSetUpdateSurge feature gate. (#96441, @smarterclayton) [SIG Apps and Testing]
• Enable SPDY pings to keep connections alive, so that kubectl exec and kubectl portforward won't be interrupted. (#97083, @knight42) [SIG API Machinery and CLI]
• FieldManager no longer owns fields that get reset before the object is persisted (e.g. "status wiping"). (#99661, @kevindelgado) [SIG API Machinery, Auth and Testing]
• Fixes server-side apply for APIService resources. (#98576, @kevindelgado)
• Generic ephemeral volumes are beta. (#99643, @pohly) [SIG API Machinery, Apps, Auth, CLI, Node, Storage and Testing]
• Hugepages request values are limited to integer multiples of the page size. (#98515, @lala123912) [SIG Apps]
• Implement the GetAvailableResources in the podresources API. (#95734, @fromanirh) [SIG Instrumentation, Node and Testing]
• IngressClass resource can now reference a resource in a specific namespace for implementation-specific configuration (previously only Cluster-level resources were allowed). This feature can be enabled using the IngressClassNamespacedParams feature gate. (#99275, @hbagdi)
• Jobs API has a new .spec.suspend field that can be used to suspend and resume Jobs. This is an alpha field which is only honored by servers with the SuspendJob feature gate enabled. (#98727, @adtac)
• Kubelet Graceful Node Shutdown feature graduates to Beta and enabled by default. (#99735, @bobbypage)
• Kubernetes is now built using go1.15.7 (#98363, @cpanato) [SIG Cloud Provider, Instrumentation, Node, Release and Testing]
• Namespace API objects now have a kubernetes.io/metadata.name label matching their metadata.name field to allow selecting any namespace by its name using a label selector. (#96968, @jayunit100) [SIG API Machinery, Apps, Cloud Provider, Storage and Testing]
• One new field "InternalTrafficPolicy" in Service is added. It specifies if the cluster internal traffic should be routed to all endpoints or node-local endpoints only. "Cluster" routes internal traffic to a Service to all endpoints. "Local" routes traffic to node-local endpoints only, and traffic is dropped if no node-local endpoints are ready. The default value is "Cluster". (#96600, @maplain) [SIG API Machinery, Apps and Network]
• PodDisruptionBudget API objects can now contain conditions in status. (#98127, @mortent) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Cluster Lifecycle and Instrumentation]
• PodSecurityPolicy only stores "generic" as allowed volume type if the GenericEphemeralVolume feature gate is enabled (#98918, @pohly) [SIG Auth and Security]
• Promote CronJobs to batch/v1 (#99423, @soltysh) [SIG API Machinery, Apps, CLI and Testing]
• Promote Immutable Secrets/ConfigMaps feature to Stable. This allows to set immutable field in Secret or ConfigMap object to mark their contents as immutable. (#97615, @wojtek-t) [SIG Apps, Architecture, Node and Testing]
• Remove support for building Kubernetes with bazel. (#99561, @BenTheElder) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scalability, Scheduling, Storage, Testing and Windows]
• Scheduler extender filter interface now can report unresolvable failed nodes in the new field FailedAndUnresolvableNodes of ExtenderFilterResult struct. Nodes in this map will be skipped in the preemption phase. (#92866, @cofyc) [SIG Scheduling]
• Services can specify loadBalancerClass to use a custom load balancer (#98277, @XudongLiuHarold)
• Storage capacity tracking (= the CSIStorageCapacity feature) graduates to Beta and enabled by default, storage.k8s.io/v1alpha1/VolumeAttachment and storage.k8s.io/v1alpha1/CSIStorageCapacity objects are deprecated (#99641, @pohly)
• Support for Indexed Job: a Job that is considered completed when Pods associated to indexes from 0 to (.spec.completions-1) have succeeded. (#98812, @alculquicondor) [SIG Apps and CLI]
• The BoundServiceAccountTokenVolume feature has been promoted to beta, and enabled by default.
  • This changes the tokens provided to containers at /var/run/secrets/kubernetes.io/serviceaccount/token to be time-limited, auto-refreshed, and invalidated when the containing pod is deleted.
  • Clients should reload the token from disk periodically (once per minute is recommended) to ensure they continue to use a valid token. k8s.io/client-go version v11.0.0+ and v0.15.0+ reload tokens automatically.
  • By default, injected tokens are given an extended lifetime so they remain valid even after a new refreshed token is provided. The metric serviceaccount_stale_tokens_total can be used to monitor for workloads that are depending on the extended lifetime and are continuing to use tokens even after a refreshed token is provided to the container. If that metric indicates no existing workloads are depending on extended lifetimes, injected token lifetime can be shortened to 1 hour by starting kube-apiserver with --service-account-extend-token-expiration=false. (#95667, @zshihang) [SIG API Machinery, Auth, Cluster Lifecycle and Testing]
• The EndpointSlice Controllers are now GA. The EndpointSliceController will not populate the deprecatedTopology field and will only provide topology information through the zone and nodeName fields. (#99870, @swetharepakula)
• The Endpoints controller will now set the endpoints.kubernetes.io/over-capacity annotation to "warning" when an Endpoints resource contains more than 1000 addresses. In a future release, the controller will truncate Endpoints that exceed this limit. The EndpointSlice API can be used to support significantly larger number of addresses. (#99975, @robscott) [SIG Apps and Network]
• The PodDisruptionBudget API has been promoted to policy/v1 with no schema changes. The only functional change is that an empty selector ({}) written to a policy/v1 PodDisruptionBudget now selects all pods in the namespace. The behavior of the policy/v1beta1 API remains unchanged. The policy/v1beta1 PodDisruptionBudget API is deprecated and will no longer be served in 1.25+. (#99290, @mortent) [SIG API Machinery, Apps, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Scheduling and Testing]
• The EndpointSlice API is now GA. The EndpointSlice topology field has been removed from the GA API and will be replaced by a new per Endpoint Zone field. If the topology field was previously used, it will be converted into an annotation in the v1 Resource. The discovery.k8s.io/v1alpha1 API is removed. (#99662, @swetharepakula)
• The controller.kubernetes.io/pod-deletion-cost annotation can be set to offer a hint on the cost of deleting a Pod compared to other pods belonging to the same ReplicaSet. Pods with lower deletion cost are deleted first. This is an alpha feature. (#99163, @ahg-g)
• The kube-apiserver now resets managedFields that got corrupted by a mutating admission controller. (#98074, @kwiesmueller)
• Topology Aware Hints are now available in alpha and can be enabled with the TopologyAwareHints feature gate. (#99522, @robscott) [SIG API Machinery, Apps, Auth, Instrumentation, Network and Testing]
• Users might specify the kubectl.kubernetes.io/default-exec-container annotation in a Pod to preselect container for kubectl commands. (#97099, @pacoxu) [SIG CLI]

Feature

• A client-go metric, rest_client_exec_plugin_call_total, has been added to track total calls to client-go credential plugins. (#98892, @ankeesler) [SIG API Machinery, Auth, Cluster Lifecycle and Instrumentation]
• A new histogram metric to track the time it took to delete a job by the TTLAfterFinished controller (#98676, @ahg-g)
• AWS cloud provider supports auto-discovering subnets without any kubernetes.io/cluster/<clusterName> tags. It also supports additional service annotation service.beta.kubernetes.io/aws-load-balancer-subnets to manually configure the subnets. (#97431, @kishorj)
• Aborting the drain command in a list of nodes will be deprecated. The new behavior will make the drain command go through all nodes even if one or more nodes failed during the drain. For now, users can try such experience by enabling --ignore-errors flag. (#98203, @yuzhiquan)
• Add --permit-address-sharing flag to kube-apiserver to listen with SO_REUSEADDR. While allowing to listen on wildcard IPs like 0.0.0.0 and specific IPs in parallel, it avoids waiting for the kernel to release socket in TIME_WAIT state, and hence, considerably reducing kube-apiserver restart times under certain conditions. (#93861, @sttts)
• Add csi_operations_seconds metric on kubelet that exposes CSI operations duration and status for node CSI operations. (#98979, @Jiawei0227) [SIG Instrumentation and Storage]
• Add migrated field into storage_operation_duration_seconds metric (#99050, @Jiawei0227) [SIG Apps, Instrumentation and Storage]
• Add flag --lease-reuse-duration-seconds for kube-apiserver to config etcd lease reuse duration. (#97009, @lingsamuel) [SIG API Machinery and Scalability]
• Add metric etcd_lease_object_counts for kube-apiserver to observe max objects attached to a single etcd lease. (#97480, @lingsamuel) [SIG API Machinery, Instrumentation and Scalability]
• Add support to generate client-side binaries for new darwin/arm64 platform (#97743, @dims) [SIG Release and Testing]
• Added ephemeral_volume_controller_create[_failures]_total counters to kube-controller-manager metrics (#99115, @pohly) [SIG API Machinery, Apps, Cluster Lifecycle, Instrumentation and Storage]
• Added support for installing arm64 node artifacts. (#99242, @liu-cong)
• Adds alpha feature VolumeCapacityPriority which makes the scheduler prioritize nodes based on the best matching size of statically provisioned PVs across multiple topologies. (#96347, @cofyc) [SIG Apps, Network, Scheduling, Storage and Testing]
• Adds the ability to pass --strict-transport-security-directives to the kube-apiserver to set the HSTS header appropriately. Be sure you understand the consequences to browsers before setting this field. (#96502, @249043822) [SIG Auth]
• Adds two new metrics to cronjobs, a histogram to track the time difference when a job is created and the expected time when it should be created, as well as a gauge for the missed schedules of a cronjob (#99341, @alaypatel07)
• Alpha implementation of Kubectl Command Headers: SIG CLI KEP 859 enabled when KUBECTL_COMMAND_HEADERS environment variable set on the client command line. (#98952, @seans3)
• Base-images: Update to debian-iptables:buster-v1.4.0
  • Uses iptables 1.8.5
  • base-images: Update to debian-base:buster-v1.3.0
  • cluster/images/etcd: Build etcd:3.4.13-2 image
    • Uses debian-base:buster-v1.3.0 (#98401, @pacoxu) [SIG Testing]
• CRIContainerLogRotation graduates to GA and unconditionally enabled. (#99651, @umohnani8)
• Component owner can configure the allowlist of metric label with flag '--allow-metric-labels'. (#99385, @YoyinZyc) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Release]
• Component owner can configure the allowlist of metric label with flag '--allow-metric-labels'. (#99738, @YoyinZyc) [SIG API Machinery, Cluster Lifecycle and Instrumentation]
• EmptyDir memory backed volumes are sized as the the minimum of pod allocatable memory on a host and an optional explicit user provided value. (#100319, @derekwaynecarr) [SIG Node]
• Enables Kubelet to check volume condition and log events to corresponding pods. (#99284, @fengzixu) [SIG Apps, Instrumentation, Node and Storage]
• EndpointSliceNodeName graduates to GA and thus will be unconditionally enabled -- NodeName will always be available in the v1beta1 API. (#99746, @swetharepakula)
• Export NewDebuggingRoundTripper function and DebugLevel options in the k8s.io/client-go/transport package. (#98324, @atosatto)
• Kube-proxy iptables: new metric sync_proxy_rules_iptables_total that exposes the number of rules programmed per table in each iteration (#99653, @aojea) [SIG Instrumentation and Network]
• Kube-scheduler now logs plugin scoring summaries at --v=4 (#99411, @damemi) [SIG Scheduling]
• Kubeadm now includes CoreDNS v1.8.0. (#96429, @rajansandeep) [SIG Cluster Lifecycle]
• Kubeadm: IPv6DualStack feature gate graduates to Beta and enabled by default (#99294, @pacoxu)
• Kubeadm: a warning to user as ipv6 site-local is deprecated (#99574, @pacoxu) [SIG Cluster Lifecycle and Network]
• Kubeadm: add support for certificate chain validation. When using kubeadm in external CA mode, this allows an intermediate CA to be used to sign the certificates. The intermediate CA certificate must be appended to each signed certificate for this to work correctly. (#97266, @robbiemcmichael) [SIG Cluster Lifecycle]
• Kubeadm: amend the node kernel validation to treat CGROUP_PIDS, FAIR_GROUP_SCHED as required and CFS_BANDWIDTH, CGROUP_HUGETLB as optional (#96378, @neolit123) [SIG Cluster Lifecycle and Node]
• Kubeadm: apply the "node.kubernetes.io/exclude-from-external-load-balancers" label on control plane nodes during "init", "join" and "upgrade" to preserve backwards compatibility with the lagacy LB mode where nodes labeled as "master" where excluded. To opt-out you can remove the label from a node. See #97543 and the linked KEP for more details. (#98269, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: if the user has customized their image repository via the kubeadm configuration, pass the custom pause image repository and tag to the kubelet via --pod-infra-container-image not only for Docker but for all container runtimes. This flag tells the kubelet that it should not garbage collect the image. (#99476, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: perform pre-flight validation on host/node name upon kubeadm init and kubeadm join, showing warnings on non-compliant names (#99194, @pacoxu)
• Kubectl version changed to write a warning message to stderr if the client and server version difference exceeds the supported version skew of +/-1 minor version. (#98250, @brianpursley) [SIG CLI]
• Kubectl: Add --use-protocol-buffers flag to kubectl top pods and nodes. (#96655, @serathius)
• Kubectl: kubectl get will omit managed fields by default now. Users could set --show-managed-fields to true to show managedFields when the output format is either json or yaml. (#96878, @knight42) [SIG CLI and Testing]
• Kubectl: a Pod can be preselected as default container using kubectl.kubernetes.io/default-container annotation (#99833, @mengjiao-liu)
• Kubectl: add bash-completion for comma separated list on kubectl get (#98301, @phil9909)
• Kubernetes is now built using go1.15.8 (#98834, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]
• Kubernetes is now built with Golang 1.16 (#98572, @justaugustus) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Release and Testing]
• Kubernetes is now built with Golang 1.16.1 (#100106, @justaugustus) [SIG Cloud Provider, Instrumentation, Release and Testing]
• Metrics can now be disabled explicitly via a command line flag (i.e. '--disabled-metrics=metric1,metric2') (#99217, @logicalhan)
• New admission controller DenyServiceExternalIPs is available. Clusters which do not need the Service externalIPs feature should enable this controller and be more secure. (#97395, @thockin)
• Overall, enable the feature of PreferNominatedNode will improve the performance of scheduling where preemption might frequently happen, but in theory, enable the feature of PreferNominatedNode, the pod might not be scheduled to the best candidate node in the cluster. (#93179, @chendave) [SIG Scheduling and Testing]
• Persistent Volumes formatted with the btrfs filesystem will now automatically resize when expanded. (#99361, @Novex) [SIG Storage]
• Port the devicemanager to Windows node to allow device plugins like directx (#93285, @aarnaud) [SIG Node, Testing and Windows]
• Removes cAdvisor JSON metrics (/stats/container, /stats//, /stats////) from the kubelet. (#99236, @pacoxu)
• Rename metrics etcd_object_counts to apiserver_storage_object_counts and mark it as stable. The original etcd_object_counts metrics name is marked as "Deprecated" and will be removed in the future. (#99785, @erain) [SIG API Machinery, Instrumentation and Testing]
• Sysctls graduates to General Availability and thus unconditionally enabled. (#99158, @wgahnagl)
• The Kubernetes pause image manifest list now contains an image for Windows Server 20H2. (#97322, @claudiubelu) [SIG Windows]
• The NodeAffinity plugin implements the PreFilter extension, offering enhanced performance for Filter. (#99213, @AliceZhang2016) [SIG Scheduling]
• The CronJobControllerV2 feature flag graduates to Beta and set to be enabled by default. (#98878, @soltysh)
• The EndpointSlice mirroring controller mirrors endpoints annotations and labels to the generated endpoint slices, it also ensures that updates on any of these fields are mirrored. The well-known annotation endpoints.kubernetes.io/last-change-trigger-time is skipped and not mirrored. (#98116, @aojea)
• The RunAsGroup feature has been promoted to GA in this release. (#94641, @krmayankk) [SIG Auth and Node]
• The ServiceAccountIssuerDiscovery feature has graduated to GA, and is unconditionally enabled. The ServiceAccountIssuerDiscovery feature-gate will be removed in 1.22. (#98553, @mtaufen) [SIG API Machinery, Auth and Testing]
• The TTLAfterFinished feature flag is now beta and enabled by default (#98678, @ahg-g)
• The apimachinery util/net function used to detect the bind address ResolveBindAddress() takes into consideration global IP addresses on loopback interfaces when 1) the host has default routes, or 2) there are no global IPs on those interfaces in order to support more complex network scenarios like BGP Unnumbered RFC 5549 (#95790, @aojea) [SIG Network]
• The feature gate RootCAConfigMap graduated to GA in v1.21 and therefore will be unconditionally enabled. This flag will be removed in v1.22 release. (#98033, @zshihang)
• The pause image upgraded to v3.4.1 in kubelet and kubeadm for both Linux and Windows. (#98205, @pacoxu)
• Update pause container to run as pseudo user and group 65535:65535. This implies the release of version 3.5 of the container images. (#97963, @saschagrunert) [SIG CLI, Cloud Provider, Cluster Lifecycle, Node, Release, Security and Testing]
• Update the latest validated version of Docker to 20.10 (#98977, @neolit123) [SIG CLI, Cluster Lifecycle and Node]
• Upgrade node local dns to 1.17.0 for better IPv6 support (#99749, @pacoxu) [SIG Cloud Provider and Network]
• Upgrades IPv6Dualstack to Beta and turns it on by default. New clusters or existing clusters are not be affected until an actor starts adding secondary Pods and service CIDRS CLI flags as described here: IPv4/IPv6 Dual-stack (#98969, @khenidak)
• Users might specify the kubectl.kubernetes.io/default-container annotation in a Pod to preselect container for kubectl commands. (#99581, @mengjiao-liu) [SIG CLI]
• When downscaling ReplicaSets, ready and creation timestamps are compared in a logarithmic scale. (#99212, @damemi) [SIG Apps and Testing]
• When the kubelet is watching a ConfigMap or Secret purely in the context of setting environment variables for containers, only hold that watch for a defined duration before cancelling it. This change reduces the CPU and memory usage of the kube-apiserver in large clusters. (#99393, @chenyw1990) [SIG API Machinery, Node and Testing]
• WindowsEndpointSliceProxying feature gate has graduated to beta and is enabled by default. This means kube-proxy will read from EndpointSlices instead of Endpoints on Windows by default. (#99794, @robscott) [SIG Network]
• kubectl wait ensures that observedGeneration >= generation to prevent stale state reporting. An example scenario can be found on CRD updates. (#97408, @KnicKnic)

Documentation

• Azure file migration graduates to beta, with CSIMigrationAzureFile flag off by default as it requires installation of AzureFile CSI Driver. Users should enable CSIMigration and CSIMigrationAzureFile features and install the AzureFile CSI Driver to avoid disruption to existing Pod and PVC objects at that time. Azure File CSI driver does not support using same persistent volume with different fsgroups. When CSI migration is enabled for azurefile driver, such case is not supported. (there is a case we support where volume is mounted with 0777 and then it readable/writable by everyone) (#96293, @andyzhangx)
• Official support to build kubernetes with docker-machine / remote docker is removed. This change does not affect building kubernetes with docker locally. (#97935, @adeniyistephen) [SIG Release and Testing]
• Set kubelet option --volume-stats-agg-period to negative value to disable volume calculations. (#96675, @pacoxu) [SIG Node]

Failing Test

• Escape the special characters like [, ] and that exist in vsphere windows path (#98830, @liyanhui1228) [SIG Storage and Windows]
• Kube-proxy: fix a bug on UDP NodePort Services where stale connection tracking entries may blackhole the traffic directed to the NodePort (#98305, @aojea)
• Kubelet: fixes a bug in the HostPort dockershim implementation that caused the conformance test "HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol" to fail. (#98755, @aojea) [SIG Cloud Provider, Network and Node]

Bug or Regression

• AcceleratorStats will be available in the Summary API of kubelet when cri_stats_provider is used. (#96873, @ruiwen-zhao) [SIG Node]
• All data is no longer automatically deleted when a failure is detected during creation of the volume data file on a CSI volume. Now only the data file and volume path is removed. (#96021, @huffmanca)
• Clean ReplicaSet by revision instead of creation timestamp in deployment controller (#97407, @waynepeking348) [SIG Apps]
• Cleanup subnet in frontend IP configs to prevent huge subnet request bodies in some scenarios. (#98133, @nilo19) [SIG Cloud Provider]
• Client-go exec credential plugins will pass stdin only when interactive terminal is detected on stdin. This fixes a bug where previously it was checking if stdout is an interactive terminal. (#99654, @ankeesler)
• Cloud-controller-manager: routes controller should not depend on --allocate-node-cidrs (#97029, @andrewsykim) [SIG Cloud Provider and Testing]
• Cluster Autoscaler version bump to v1.20.0 (#97011, @towca)
• Creating a PVC with DataSource should fail for non-CSI plugins. (#97086, @xing-yang) [SIG Apps and Storage]
• EndpointSlice controller is now less likely to emit FailedToUpdateEndpointSlices events. (#99345, @robscott) [SIG Apps and Network]
• EndpointSlice controllers are less likely to create duplicate EndpointSlices. (#100103, @robscott) [SIG Apps and Network]
• EndpointSliceMirroring controller is now less likely to emit FailedToUpdateEndpointSlices events. (#99756, @robscott) [SIG Apps and Network]
• Ensure all vSphere nodes are are tracked by volume attach-detach controller (#96689, @gnufied)
• Ensure empty string annotations are copied over in rollbacks. (#94858, @waynepeking348)
• Ensure only one LoadBalancer rule is created when HA mode is enabled (#99825, @feiskyer) [SIG Cloud Provider]
• Ensure that client-go's EventBroadcaster is safe (non-racy) during shutdown. (#95664, @DirectXMan12) [SIG API Machinery]
• Explicitly pass KUBE_BUILD_CONFORMANCE=y in package-tarballs to reenable building the conformance tarballs. (#100571, @puerco)
• Fix Azure file migration e2e test failure when CSIMigration is turned on. (#97877, @andyzhangx)
• Fix CSI-migrated inline EBS volumes failing to mount if their volumeID is prefixed by aws:// (#96821, @wongma7) [SIG Storage]
• Fix CVE-2020-8555 for Gluster client connections. (#97922, @liggitt) [SIG Storage]
• Fix NPE in ephemeral storage eviction (#98261, @wzshiming) [SIG Node]
• Fix PermissionDenied issue on SMB mount for Windows (#99550, @andyzhangx)
• Fix bug that would let the Horizontal Pod Autoscaler scale down despite at least one metric being unavailable/invalid (#99514, @mikkeloscar) [SIG Apps and Autoscaling]
• Fix cgroup handling for systemd with cgroup v2 (#98365, @odinuge) [SIG Node]
• Fix counting error in service/nodeport/loadbalancer quota check (#97451, @pacoxu) [SIG API Machinery, Network and Testing]
• Fix errors when accessing Windows container stats for Dockershim (#98510, @jsturtevant) [SIG Node and Windows]
• Fix kube-proxy container image architecture for non amd64 images. (#98526, @saschagrunert)
• Fix missing cadvisor machine metrics. (#97006, @lingsamuel) [SIG Node]
• Fix nil VMSS name when setting service to auto mode (#97366, @nilo19) [SIG Cloud Provider]
• Fix privileged config of Pod Sandbox which was previously ignored. (#96877, @xeniumlee)
• Fix the panic when kubelet registers if a node object already exists with no Status.Capacity or Status.Allocatable (#95269, @SataQiu) [SIG Node]
• Fix the regression with the slow pods termination. Before this fix pods may take an additional time to terminate - up to one minute. Reversing the change that ensured that CNI resources cleaned up when the pod is removed on API server. (#97980, @SergeyKanzhelev) [SIG Node]
• Fix to recover CSI volumes from certain dangling attachments (#96617, @yuga711) [SIG Apps and Storage]
• Fix: azure file latency issue for metadata-heavy workloads (#97082, @andyzhangx) [SIG Cloud Provider and Storage]
• Fixed Cinder volume IDs on OpenStack Train (#96673, @jsafrane) [SIG Cloud Provider]
• Fixed FibreChannel volume plugin corrupting filesystems on detach of multipath volumes. (#97013, @jsafrane) [SIG Storage]
• Fixed a bug in kubelet that will saturate CPU utilization after containerd got restarted. (#97174, @hanlins) [SIG Node]
• Fixed a bug that causes smaller number of conntrack-max being used under CPU static policy. (#99225, @xh4n3) (#99613, @xh4n3) [SIG Network]
• Fixed a bug that on k8s nodes, when the policy of INPUT chain in filter table is not ACCEPT, healthcheck nodeport would not work. Added iptables rules to allow healthcheck nodeport traffic. (#97824, @hanlins) [SIG Network]
• Fixed a bug that the kubelet cannot start on BtrfS. (#98042, @gjkim42) [SIG Node]
• Fixed a race condition on API server startup ensuring previously created webhook configurations are effective before the first write request is admitted. (#95783, @roycaihw) [SIG API Machinery]
• Fixed an issue with garbage collection failing to clean up namespaced children of an object also referenced incorrectly by cluster-scoped children (#98068, @liggitt) [SIG API Machinery and Apps]
• Fixed authentication_duration_seconds metric scope. Previously, it included whole apiserver request duration which yields inaccurate results. (#99944, @marseel)
• Fixed bug in CPUManager with race on container map access (#97427, @klueska) [SIG Node]
• Fixed bug that caused cAdvisor to incorrectly detect single-socket multi-NUMA topology. (#99315, @iwankgb) [SIG Node]
• Fixed cleanup of block devices when /var/lib/kubelet is a symlink. (#96889, @jsafrane) [SIG Storage]
• Fixed no effect namespace when exposing deployment with --dry-run=client. (#97492, @masap) [SIG CLI]
• Fixed provisioning of Cinder volumes migrated to CSI when StorageClass with AllowedTopologies was used. (#98311, @jsafrane) [SIG Storage]
• Fixes a bug of identifying the correct containerd process. (#97888, @pacoxu)
• Fixes add-on manager leader election to use leases instead of endpoints, similar to what kube-controller-manager does in 1.20 (#98968, @liggitt)
• Fixes connection errors when using --volume-host-cidr-denylist or --volume-host-allow-local-loopback (#98436, @liggitt) [SIG Network and Storage]
• Fixes problem where invalid selector on PodDisruptionBudget leads to a nil pointer dereference that causes the Controller manager to crash loop. (#98750, @mortent)
• Fixes spurious errors about IPv6 in kube-proxy logs on nodes with IPv6 disabled. (#99127, @danwinship)
• Fixing a bug where a failed node may not have the NoExecute taint set correctly (#96876, @howieyuen) [SIG Apps and Node]
• GCE Internal LoadBalancer sync loop will now release the ILB IP address upon sync failure. An error in ILB forwarding rule creation will no longer leak IP addresses. (#97740, @prameshj) [SIG Cloud Provider and Network]
• Ignore update pod with no new images in alwaysPullImages admission controller (#96668, @pacoxu) [SIG Apps, Auth and Node]
• Improve speed of vSphere PV provisioning and reduce number of API calls (#100054, @gnufied) [SIG Cloud Provider and Storage]
• KUBECTL_EXTERNAL_DIFF now accepts equal sign for additional parameters. (#98158, @dougsland) [SIG CLI]
• Kube-apiserver: an update of a pod with a generic ephemeral volume dropped that volume if the feature had been disabled since creating the pod with such a volume (#99446, @pohly) [SIG Apps, Node and Storage]
• Kube-proxy: remove deprecated --cleanup-ipvs flag of kube-proxy, and make --cleanup flag always to flush IPVS (#97336, @maaoBit) [SIG Network]
• Kubeadm installs etcd v3.4.13 when creating cluster v1.19 (#97244, @pacoxu)
• Kubeadm: Fixes a kubeadm upgrade bug that could cause a custom CoreDNS configuration to be replaced with the default. (#97016, @rajansandeep) [SIG Cluster Lifecycle]
• Kubeadm: Some text in the kubeadm upgrade plan output has changed. If you have scripts or other automation that parses this output, please review these changes and update your scripts to account for the new output. (#98728, @stmcginnis) [SIG Cluster Lifecycle]
• Kubeadm: fix a bug in the host memory detection code on 32bit Linux platforms (#97403, @abelbarrera15) [SIG Cluster Lifecycle]
• Kubeadm: fix a bug where "kubeadm join" would not properly handle missing names for existing etcd members. (#97372, @ihgann) [SIG Cluster Lifecycle]
• Kubeadm: fix a bug where "kubeadm upgrade" commands can fail if CoreDNS v1.8.0 is installed. (#97919, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: fix a bug where external credentials in an existing admin.conf prevented the CA certificate to be written in the cluster-info ConfigMap. (#98882, @kvaps) [SIG Cluster Lifecycle]
• Kubeadm: get k8s CI version markers from k8s infra bucket (#98836, @hasheddan) [SIG Cluster Lifecycle and Release]
• Kubeadm: skip validating pod subnet against node-cidr-mask when allocate-node-cidrs is set to be false (#98984, @SataQiu) [SIG Cluster Lifecycle]
• Kubectl logs: --ignore-errors is now honored by all containers, maintaining consistency with parallelConsumeRequest behavior. (#97686, @wzshiming)
• Kubectl-convert: Fix no kind "Ingress" is registered for version error (#97754, @wzshiming)
• Kubectl: Fixed panic when describing an ingress backend without an API Group (#100505, @lauchokyip) [SIG CLI]
• Kubelet now cleans up orphaned volume directories automatically (#95301, @lorenz) [SIG Node and Storage]
• Kubelet.exe on Windows now checks that the process running as administrator and the executing user account is listed in the built-in administrators group. This is the equivalent to checking the process is running as uid 0. (#96616, @perithompson) [SIG Node and Windows]
• Kubelet: Fix kubelet from panic after getting the wrong signal (#98200, @wzshiming) [SIG Node]
• Kubelet: Fix repeatedly acquiring the inhibit lock (#98088, @wzshiming) [SIG Node]
• Kubelet: Fixed the bug of getting the number of cpu when the number of cpu logical processors is more than 64 in windows (#97378, @hwdef) [SIG Node and Windows]
• Limits lease to have 1000 maximum attached objects. (#98257, @lingsamuel)
• Mitigate CVE-2020-8555 for kube-up using GCE by preventing local loopback folume hosts. (#97934, @mattcary) [SIG Cloud Provider and Storage]
• On single-stack configured (IPv4 or IPv6, but not both) clusters, Services which are both headless (no clusterIP) and selectorless (empty or undefined selector) will report ipFamilyPolicy RequireDualStack and will have entries in ipFamilies[] for both IPv4 and IPv6. This is a change from alpha, but does not have any impact on the manually-specified Endpoints and EndpointSlices for the Service. (#99555, @thockin) [SIG Apps and Network]
• Performance regression #97685 has been fixed. (#97860, @MikeSpreitzer) [SIG API Machinery]
• Pod Log stats for windows now reports metrics (#99221, @jsturtevant) [SIG Node, Storage, Testing and Windows]
• Pod status updates faster when reacting on probe results. The first readiness probe will be called faster when startup probes succeeded, which will make Pod status as ready faster. (#98376, @matthyx)
• Readjust kubelet_containers_per_pod_count buckets to only show metrics greater than 1. (#98169, @wawa0210)
• Remove CSI topology from migrated in-tree gcepd volume. (#97823, @Jiawei0227) [SIG Cloud Provider and Storage]
• Requests with invalid timeout parameters in the request URL now appear in the audit log correctly. (#96901, @tkashem) [SIG API Machinery and Testing]
• Resolve a "concurrent map read and map write" crashing error in the kubelet (#95111, @choury) [SIG Node]
• Resolves spurious Failed to list *v1.Secret or Failed to list *v1.ConfigMap messages in kubelet logs. (#99538, @liggitt) [SIG Auth and Node]
• ResourceQuota of an entity now inclusively calculate Pod overhead (#99600, @gjkim42)
• Return zero time (midnight on Jan. 1, 1970) instead of negative number when reporting startedAt and finishedAt of the not started or a running Pod when using dockershim as a runtime. (#99585, @Iceber)
• Reverts breaking change to inline AzureFile volumes; referenced secrets are now searched for in the same namespace as the pod as in previous releases. (#100563, @msau42)
• Scores from InterPodAffinity have stronger differentiation. (#98096, @leileiwan) [SIG Scheduling]
• Specifying the KUBE_TEST_REPO environment variable when e2e tests are executed will instruct the test infrastructure to load that image from a location within the specified repo, using a predefined pattern. (#93510, @smarterclayton) [SIG Testing]
• Static pods will be deleted gracefully. (#98103, @gjkim42) [SIG Node]
• Sync node status during kubelet node shutdown. Adds an pod admission handler that rejects new pods when the node is in progress of shutting down. (#98005, @wzshiming) [SIG Node]
• The calculation of pod UIDs for static pods has changed to ensure each static pod gets a unique value - this will cause all static pod containers to be recreated/restarted if an in-place kubelet upgrade from 1.20 to 1.21 is performed. Note that draining pods before upgrading the kubelet across minor versions is the supported upgrade path. (#87461, @bboreham) [SIG Node]
• The maximum number of ports allowed in EndpointSlices has been increased from 100 to 20,000 (#99795, @robscott) [SIG Network]
• Truncates a message if it hits the NoteLengthLimit when the scheduler records an event for the pod that indicates the pod has failed to schedule. (#98715, @carlory)
• Updated k8s.gcr.io/ingress-gce-404-server-with-metrics-amd64 to a version that serves /metrics endpoint on a non-default port. (#97621, @vbannai) [SIG Cloud Provider]
• Updates the commands `
  • kubectl kustomize {arg}
  • kubectl apply -k {arg} `to use same code as kustomize CLI v4.0.5 (#98946, @monopole)
• Use force unmount for NFS volumes if regular mount fails after 1 minute timeout (#96844, @gnufied) [SIG Storage]
• Use network.Interface.VirtualMachine.ID to get the binded VM Skip standalone VM when reconciling LoadBalancer (#97635, @nilo19) [SIG Cloud Provider]
• Using exec auth plugins with kubectl no longer results in warnings about constructing many client instances from the same exec auth config. (#97857, @liggitt) [SIG API Machinery and Auth]
• When a CNI plugin returns dual-stack pod IPs, kubelet will now try to respect the "primary IP family" of the cluster by picking a primary pod IP of the same family as the (primary) node IP, rather than assuming that the CNI plugin returned the IPs in the order the administrator wanted (since some CNI plugins don't allow configuring this). (#97979, @danwinship) [SIG Network and Node]
• When dynamically provisioning Azure File volumes for a premium account, the requested size will be set to 100GB if the request is initially lower than this value to accommodate Azure File requirements. (#99122, @huffmanca) [SIG Cloud Provider and Storage]
• When using Containerd on Windows, the C:\Windows\System32\drivers\etc\hosts file will now be managed by kubelet. (#83730, @claudiubelu)
• VolumeBindingArgs now allow BindTimeoutSeconds to be set as zero, while the value zero indicates no waiting for the checking of volume binding operation. (#99835, @chendave) [SIG Scheduling and Storage]
• kubectl exec and kubectl attach now honor the --quiet flag which suppresses output from the local binary that could be confused by a script with the remote command output (all non-failure output is hidden). In addition, print inline with exec and attach the list of alternate containers when we default to the first spec.container. (#99004, @smarterclayton) [SIG CLI]

Other (Cleanup or Flake)

• APIs for kubelet annotations and labels from k8s.io/kubernetes/pkg/kubelet/apis are now moved under k8s.io/kubelet/pkg/apis/ (#98931, @michaelbeaumont)
• Apiserver_request_duration_seconds is promoted to stable status. (#99925, @logicalhan) [SIG API Machinery, Instrumentation and Testing]
• Bump github.com/Azure/go-autorest/autorest to v0.11.12 (#97033, @patrickshan) [SIG API Machinery, CLI, Cloud Provider and Cluster Lifecycle]
• Clients required to use go1.15.8+ or go1.16+ if kube-apiserver has the goaway feature enabled to avoid unexpected data race condition. (#98809, @answer1991)
• Delete deprecated service.beta.kubernetes.io/azure-load-balancer-mixed-protocols mixed procotol annotation in favor of the MixedProtocolLBService feature (#97096, @nilo19) [SIG Cloud Provider]
• EndpointSlice generation is now incremented when labels change. (#99750, @robscott) [SIG Network]
• Featuregate AllowInsecureBackendProxy graduates to GA and unconditionally enabled. (#99658, @deads2k)
• Increase timeout for pod lifecycle test to reach pod status=ready (#96691, @hh)
• Increased CSINodeIDMaxLength from 128 bytes to 192 bytes. (#98753, @Jiawei0227)
• Kube-apiserver: The OIDC authenticator no longer waits 10 seconds before attempting to fetch the metadata required to verify tokens. (#97693, @enj) [SIG API Machinery and Auth]
• Kube-proxy: Traffic from the cluster directed to ExternalIPs is always sent directly to the Service. (#96296, @aojea) [SIG Network and Testing]
• Kubeadm: change the default image repository for CI images from 'gcr.io/kubernetes-ci-images' to 'gcr.io/k8s-staging-ci-images' (#97087, @SataQiu) [SIG Cluster Lifecycle]
• Kubectl: The deprecated kubectl alpha debug command is removed. Use kubectl debug instead. (#98111, @pandaamanda) [SIG CLI]
• Kubelet command line flags related to dockershim are now showing deprecation message as they will be removed along with dockershim in future release. (#98730, @dims)
• Official support to build kubernetes with docker-machine / remote docker is removed. This change does not affect building kubernetes with docker locally. (#97618, @jherrera123) [SIG Release and Testing]
• Process start time on Windows now uses current process information (#97491, @jsturtevant) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Windows]
• Resolves flakes in the Ingress conformance tests due to conflicts with controllers updating the Ingress object (#98430, @liggitt) [SIG Network and Testing]
• The AttachVolumeLimit feature gate (GA since v1.17) has been removed and now unconditionally enabled. (#96539, @ialidzhikov)
• The CSINodeInfo feature gate that is GA since v1.17 is unconditionally enabled, and can no longer be specified via the --feature-gates argument. (#96561, @ialidzhikov) [SIG Apps, Auth, Scheduling, Storage and Testing]
• The apiserver_request_total metric is promoted to stable status and no longer has a content-type dimensions, so any alerts/charts which presume the existence of this will fail. This is however, unlikely to be the case since it was effectively an unbounded dimension in the first place. (#99788, @logicalhan)
• The default delegating authorization options now allow unauthenticated access to healthz, readyz, and livez. A system:masters user connecting to an authz delegator will not perform an authz check. (#98325, @deads2k) [SIG API Machinery, Auth, Cloud Provider and Scheduling]
• The deprecated feature gates CSIDriverRegistry, BlockVolume and CSIBlockVolume are now unconditionally enabled and can no longer be specified in component invocations. (#98021, @gavinfish) [SIG Storage]
• The deprecated feature gates RotateKubeletClientCertificate, AttachVolumeLimit, VolumePVCDataSource and EvenPodsSpread are now unconditionally enabled and can no longer be specified in component invocations. (#97306, @gavinfish) [SIG Node, Scheduling and Storage]
• The e2e suite can be instructed not to wait for pods in kube-system to be ready or for all nodes to be ready by passing --allowed-not-ready-nodes=-1 when invoking the e2e.test program. This allows callers to run subsets of the e2e suite in scenarios other than perfectly healthy clusters. (#98781, @smarterclayton) [SIG Testing]
• The feature gates WindowsGMSA and WindowsRunAsUserName that are GA since v1.18 are now removed. (#96531, @ialidzhikov) [SIG Node and Windows]
• The new -gce-zones flag on the e2e.test binary instructs tests that check for information about how the cluster interacts with the cloud to limit their queries to the provided zone list. If not specified, the current behavior of asking the cloud provider for all available zones in multi zone clusters is preserved. (#98787, @smarterclayton) [SIG API Machinery, Cluster Lifecycle and Testing]
• Update cri-tools to v1.20.0 (#97967, @rajibmitra) [SIG Cloud Provider]
• Windows nodes on GCE will take longer to start due to dependencies installed at node creation time. (#98284, @pjh) [SIG Cloud Provider]
• apiserver_storage_objects (a newer version of etcd_object_counts) is promoted and marked as stable. (#100082, @logicalhan)

Uncategorized

• GCE L4 Loadbalancers now handle > 5 ports in service spec correctly. (#99595, @prameshj) [SIG Cloud Provider]
• The DownwardAPIHugePages feature is beta. Users may use the feature if all workers in their cluster are min 1.20 version. The feature will be enabled by default in all installations in 1.22. (#99610, @derekwaynecarr) [SIG Node]

Dependencies

Added

• github.com/go-errors/errors: v1.0.1
• github.com/gobuffalo/here: v0.6.0
• github.com/google/shlex: e7afc7f
• github.com/markbates/pkger: v0.17.1
• github.com/moby/spdystream: v0.2.0
• github.com/monochromegane/go-gitignore: 205db1a
• github.com/niemeyer/pretty: a10e7ca
• github.com/xlab/treeprint: a009c39
• go.starlark.net: 8dd3e2e
• golang.org/x/term: 6a3ed07
• sigs.k8s.io/kustomize/api: v0.8.5
• sigs.k8s.io/kustomize/cmd/config: v0.9.7
• sigs.k8s.io/kustomize/kustomize/v4: v4.0.5
• sigs.k8s.io/kustomize/kyaml: v0.10.15

Changed

• dmitri.shuralyov.com/gpu/mtl: 666a987 → 28db891
• github.com/Azure/go-autorest/autorest: v0.11.1 → v0.11.12
• github.com/NYTimes/gziphandler: 56545f4 → v1.1.1
• github.com/cilium/ebpf: 1c8d4c9 → v0.2.0
• github.com/container-storage-interface/spec: v1.2.0 → v1.3.0
• github.com/containerd/console: v1.0.0 → v1.0.1
• github.com/containerd/containerd: v1.4.1 → v1.4.4
• github.com/coredns/corefile-migration: v1.0.10 → v1.0.11
• github.com/creack/pty: v1.1.7 → v1.1.11
• github.com/docker/docker: bd33bbf → v20.10.2+incompatible
• github.com/go-logr/logr: v0.2.0 → v0.4.0
• github.com/go-openapi/spec: v0.19.3 → v0.19.5
• github.com/go-openapi/strfmt: v0.19.3 → v0.19.5
• github.com/go-openapi/validate: v0.19.5 → v0.19.8
• github.com/gogo/protobuf: v1.3.1 → v1.3.2
• github.com/golang/mock: v1.4.1 → v1.4.4
• github.com/google/cadvisor: v0.38.5 → v0.39.0
• github.com/heketi/heketi: c2e2a4a → v10.2.0+incompatible
• github.com/kisielk/errcheck: v1.2.0 → v1.5.0
• github.com/konsorten/go-windows-terminal-sequences: v1.0.3 → v1.0.2
• github.com/kr/text: v0.1.0 → v0.2.0
• github.com/mattn/go-runewidth: v0.0.2 → v0.0.7
• github.com/miekg/dns: v1.1.4 → v1.1.35
• github.com/moby/sys/mountinfo: v0.1.3 → v0.4.0
• github.com/moby/term: 672ec06 → df9cb8a
• github.com/mrunalp/fileutils: abd8a0e → v0.5.0
• github.com/olekukonko/tablewriter: a0225b3 → v0.0.4
• github.com/opencontainers/runc: v1.0.0-rc92 → v1.0.0-rc93
• github.com/opencontainers/runtime-spec: 4d89ac9 → e6143ca
• github.com/opencontainers/selinux: v1.6.0 → v1.8.0
• github.com/sergi/go-diff: v1.0.0 → v1.1.0
• github.com/sirupsen/logrus: v1.6.0 → v1.7.0
• github.com/syndtr/gocapability: d983527 → 42c35b4
• github.com/willf/bitset: d5bec33 → v1.1.11
• github.com/yuin/goldmark: v1.1.27 → v1.2.1
• golang.org/x/crypto: 7f63de1 → 5ea612d
• golang.org/x/exp: 6cc2880 → 85be41e
• golang.org/x/mobile: d2bd2a2 → e6ae53a
• golang.org/x/mod: v0.3.0 → ce943fd
• golang.org/x/net: 69a7880 → 3d97a24
• golang.org/x/sync: cd5d95a → 67f06af
• golang.org/x/sys: 5cba982 → a50acf3
• golang.org/x/time: 3af7569 → f8bda1e
• golang.org/x/tools: c1934b7 → v0.1.0
• gopkg.in/check.v1: 41f04d3 → 8fa4692
• gopkg.in/yaml.v2: v2.2.8 → v2.4.0
• gotest.tools/v3: v3.0.2 → v3.0.3
• k8s.io/gengo: 83324d8 → b6c5ce2
• k8s.io/klog/v2: v2.4.0 → v2.8.0
• k8s.io/kube-openapi: d219536 → 591a79e
• k8s.io/system-validators: v1.2.0 → v1.4.0
• sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.14 → v0.0.15
• sigs.k8s.io/structured-merge-diff/v4: v4.0.2 → v4.1.0

Removed

• github.com/codegangsta/negroni: v1.0.0
• github.com/docker/spdystream: 449fdfc
• github.com/golangplus/bytes: 45c989f
• github.com/golangplus/fmt: 2a5d6d7
• github.com/gorilla/context: v1.1.1
• github.com/kr/pty: v1.1.5
• rsc.io/quote/v3: v3.1.0
• rsc.io/sampler: v1.3.0
• sigs.k8s.io/kustomize: v2.0.3+incompatible

v1.21.0-rc.0

Downloads for v1.21.0-rc.0

Source Code

Client binaries

Server binaries

Node binaries

Changelog since v1.21.0-beta.1

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Migrated pkg/kubelet/cm/cpuset/cpuset.go to structured logging. Exit code changed from 255 to 1. (#100007, @utsavoza) [SIG Instrumentation and Node]

Changes by Kind

API Change

• Add Probe-level terminationGracePeriodSeconds field (#99375, @ehashman) [SIG API Machinery, Apps, Node and Testing]
• CSIServiceAccountToken is Beta now (#99298, @zshihang) [SIG Auth, Storage and Testing]
• Discovery.k8s.io/v1beta1 EndpointSlices are deprecated in favor of discovery.k8s.io/v1, and will no longer be served in Kubernetes v1.25. (#100472, @liggitt) [SIG Network]
• FieldManager no longer owns fields that get reset before the object is persisted (e.g. "status wiping"). (#99661, @kevindelgado) [SIG API Machinery, Auth and Testing]
• Generic ephemeral volumes are beta. (#99643, @pohly) [SIG API Machinery, Apps, Auth, CLI, Node, Storage and Testing]
• Implement the GetAvailableResources in the podresources API. (#95734, @fromanirh) [SIG Instrumentation, Node and Testing]
• The Endpoints controller will now set the endpoints.kubernetes.io/over-capacity annotation to "warning" when an Endpoints resource contains more than 1000 addresses. In a future release, the controller will truncate Endpoints that exceed this limit. The EndpointSlice API can be used to support significantly larger number of addresses. (#99975, @robscott) [SIG Apps and Network]
• The PodDisruptionBudget API has been promoted to policy/v1 with no schema changes. The only functional change is that an empty selector ({}) written to a policy/v1 PodDisruptionBudget now selects all pods in the namespace. The behavior of the policy/v1beta1 API remains unchanged. The policy/v1beta1 PodDisruptionBudget API is deprecated and will no longer be served in 1.25+. (#99290, @mortent) [SIG API Machinery, Apps, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Scheduling and Testing]
• Topology Aware Hints are now available in alpha and can be enabled with the TopologyAwareHints feature gate. (#99522, @robscott) [SIG API Machinery, Apps, Auth, Instrumentation, Network and Testing]

Feature

• Add e2e test to validate performance metrics of volume lifecycle operations (#94334, @RaunakShah) [SIG Storage and Testing]
• EmptyDir memory backed volumes are sized as the the minimum of pod allocatable memory on a host and an optional explicit user provided value. (#100319, @derekwaynecarr) [SIG Node]
• Enables Kubelet to check volume condition and log events to corresponding pods. (#99284, @fengzixu) [SIG Apps, Instrumentation, Node and Storage]
• Introduce a churn operator to scheduler perf testing framework. (#98900, @Huang-Wei) [SIG Scheduling and Testing]
• Kubernetes is now built with Golang 1.16.1 (#100106, @justaugustus) [SIG Cloud Provider, Instrumentation, Release and Testing]
• Migrated pkg/kubelet/cm/devicemanager to structured logging (#99976, @knabben) [SIG Instrumentation and Node]
• Migrated pkg/kubelet/cm/memorymanager to structured logging (#99974, @knabben) [SIG Instrumentation and Node]
• Migrated pkg/kubelet/cm/topologymanager to structure logging (#99969, @knabben) [SIG Instrumentation and Node]
• Rename metrics etcd_object_counts to apiserver_storage_object_counts and mark it as stable. The original etcd_object_counts metrics name is marked as "Deprecated" and will be removed in the future. (#99785, @erain) [SIG API Machinery, Instrumentation and Testing]
• Update pause container to run as pseudo user and group 65535:65535. This implies the release of version 3.5 of the container images. (#97963, @saschagrunert) [SIG CLI, Cloud Provider, Cluster Lifecycle, Node, Release, Security and Testing]
• Users might specify the kubectl.kubernetes.io/default-exec-container annotation in a Pod to preselect container for kubectl commands. (#99833, @mengjiao-liu) [SIG CLI]

Bug or Regression

• Add ability to skip OpenAPI handler installation to the GenericAPIServer (#100341, @kevindelgado) [SIG API Machinery]
• Count pod overhead against an entity's ResourceQuota (#99600, @gjkim42) [SIG API Machinery and Node]
• EndpointSlice controllers are less likely to create duplicate EndpointSlices. (#100103, @robscott) [SIG Apps and Network]
• Ensure only one LoadBalancer rule is created when HA mode is enabled (#99825, @feiskyer) [SIG Cloud Provider]
• Fixed a race condition on API server startup ensuring previously created webhook configurations are effective before the first write request is admitted. (#95783, @roycaihw) [SIG API Machinery]
• Fixed authentication_duration_seconds metric. Previously it included whole apiserver request duration. (#99944, @marseel) [SIG API Machinery, Instrumentation and Scalability]
• Fixes issue where inline AzueFile secrets could not be accessed from the pod's namespace. (#100563, @msau42) [SIG Storage]
• Improve speed of vSphere PV provisioning and reduce number of API calls (#100054, @gnufied) [SIG Cloud Provider and Storage]
• Kubectl: Fixed panic when describing an ingress backend without an API Group (#100505, @lauchokyip) [SIG CLI]
• Kubectl: fix case of age column in describe node (#96963, @bl-ue) (#96963, @bl-ue) [SIG CLI]
• Kubelet.exe on Windows now checks that the process running as administrator and the executing user account is listed in the built-in administrators group. This is the equivalent to checking the process is running as uid 0. (#96616, @perithompson) [SIG Node and Windows]
• Kubelet: Fixed the bug of getting the number of cpu when the number of cpu logical processors is more than 64 in windows (#97378, @hwdef) [SIG Node and Windows]
• Pass KUBE_BUILD_CONFORMANCE=y to the package-tarballs to reenable building the conformance tarballs. (#100571, @puerco) [SIG Release]
• Pod Log stats for windows now reports metrics (#99221, @jsturtevant) [SIG Node, Storage, Testing and Windows]

Other (Cleanup or Flake)

• A new storage E2E testsuite covers CSIStorageCapacity publishing if a driver opts into the test. (#100537, @pohly) [SIG Storage and Testing]
• Convert cmd/kubelet/app/server.go to structured logging (#98334, @wawa0210) [SIG Node]
• If kube-apiserver enabled goaway feature, clients required golang 1.15.8 or 1.16+ version to avoid un-expected data race issue. (#98809, @answer1991) [SIG API Machinery]
• Increased CSINodeIDMaxLength from 128 bytes to 192 bytes. (#98753, @Jiawei0227) [SIG Apps and Storage]
• Migrate pkg/kubelet/pluginmanager to structured logging (#99885, @qingwave) [SIG Node]
• Migrate pkg/kubelet/preemption/preemption.go and pkg/kubelet/logs/container_log_manager.go to structured logging (#99848, @qingwave) [SIG Node]
• Migrate pkg/kubelet/(cri) to structured logging (#99006, @yangjunmyfm192085) [SIG Node]
• Migrate pkg/kubelet/(node, pod) to structured logging (#98847, @yangjunmyfm192085) [SIG Node]
• Migrate pkg/kubelet/(volume,container) to structured logging (#98850, @yangjunmyfm192085) [SIG Node]
• Migrate pkg/kubelet/kubelet_node_status.go to structured logging (#98154, @yangjunmyfm192085) [SIG Node and Release]
• Migrate pkg/kubelet/lifecycle,oom to structured logging (#99479, @mengjiao-liu) [SIG Instrumentation and Node]
• Migrate cmd/kubelet/+ pkg/kubelet/cadvisor/cadvisor_linux.go + pkg/kubelet/cri/remote/util/util_unix.go + pkg/kubelet/images/image_manager.go to structured logging (#99994, @AfrouzMashayekhi) [SIG Instrumentation and Node]
• Migrate pkg/kubelet/cm/container_manager_linux.go and pkg/kubelet/cm/container_manager_stub.go to structured logging (#100001, @shiyajuan123) [SIG Instrumentation and Node]
• Migrate pkg/kubelet/cm/cpumanage/{topology/togit pology.go, policy_none.go, cpu_assignment.go} to structured logging (#100163, @lala123912) [SIG Instrumentation and Node]
• Migrate pkg/kubelet/cm/cpumanager/state to structured logging (#99563, @jmguzik) [SIG Instrumentation and Node]
• Migrate pkg/kubelet/config to structured logging (#100002, @AfrouzMashayekhi) [SIG Instrumentation and Node]
• Migrate pkg/kubelet/kubelet.go to structured logging (#99861, @navidshaikh) [SIG Instrumentation and Node]
• Migrate pkg/kubelet/kubeletconfig to structured logging (#100265, @ehashman) [SIG Node]
• Migrate pkg/kubelet/kuberuntime to structured logging (#99970, @krzysiekg) [SIG Instrumentation and Node]
• Migrate pkg/kubelet/prober to structured logging (#99830, @krzysiekg) [SIG Instrumentation and Node]
• Migrate pkg/kubelet/winstats to structured logging (#99855, @hexxdump) [SIG Instrumentation and Node]
• Migrate probe log messages to structured logging (#97093, @aldudko) [SIG Instrumentation and Node]
• Migrate remaining kubelet files to structured logging (#100196, @ehashman) [SIG Instrumentation and Node]
• apiserver_storage_objects (a newer version of `etcd_object_counts) is promoted and marked as stable. (#100082, @logicalhan) [SIG API Machinery, Instrumentation and Testing]

Dependencies

Added

Nothing has changed.

Changed

• github.com/cilium/ebpf: 1c8d4c9 → v0.2.0
• github.com/containerd/console: v1.0.0 → v1.0.1
• github.com/containerd/containerd: v1.4.1 → v1.4.4
• github.com/creack/pty: v1.1.9 → v1.1.11
• github.com/docker/docker: bd33bbf → v20.10.2+incompatible
• github.com/google/cadvisor: v0.38.8 → v0.39.0
• github.com/konsorten/go-windows-terminal-sequences: v1.0.3 → v1.0.2
• github.com/moby/sys/mountinfo: v0.1.3 → v0.4.0
• github.com/moby/term: 672ec06 → df9cb8a
• github.com/mrunalp/fileutils: abd8a0e → v0.5.0
• github.com/opencontainers/runc: v1.0.0-rc92 → v1.0.0-rc93
• github.com/opencontainers/runtime-spec: 4d89ac9 → e6143ca
• github.com/opencontainers/selinux: v1.6.0 → v1.8.0
• github.com/sirupsen/logrus: v1.6.0 → v1.7.0
• github.com/syndtr/gocapability: d983527 → 42c35b4
• github.com/willf/bitset: d5bec33 → v1.1.11
• gotest.tools/v3: v3.0.2 → v3.0.3
• k8s.io/klog/v2: v2.5.0 → v2.8.0
• sigs.k8s.io/structured-merge-diff/v4: v4.0.3 → v4.1.0

Removed

Nothing has changed.

v1.21.0-beta.1

Downloads for v1.21.0-beta.1

Source Code

Client binaries

Server binaries

Node binaries

Changelog since v1.21.0-beta.0

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Kubeadm: during "init" an empty cgroupDriver value in the KubeletConfiguration is now always set to "systemd" unless the user is explicit about it. This requires existing machine setups to configure the container runtime to use the "systemd" driver. Documentation on this topic can be found here: https://kubernetes.io/docs/setup/production-environment/container-runtimes/. When upgrading existing clusters / nodes using "kubeadm upgrade" the old cgroupDriver value is preserved, but in 1.22 this change will also apply to "upgrade". For more information on migrating to the "systemd" driver or remaining on the "cgroupfs" driver see: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/. (#99471, @neolit123) [SIG Cluster Lifecycle]
• Migrate pkg/kubelet/(dockershim, network) to structured logging Exit code changed from 255 to 1 (#98939, @yangjunmyfm192085) [SIG Network and Node]
• Migrate pkg/kubelet/certificate to structured logging Exit code changed from 255 to 1 (#98993, @SataQiu) [SIG Auth and Node]
• Newly provisioned PVs by EBS plugin will no longer use the deprecated "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" labels. It will use "topology.kubernetes.io/zone" and "topology.kubernetes.io/region" labels instead. (#99130, @ayberk) [SIG Cloud Provider, Storage and Testing]
• Newly provisioned PVs by OpenStack Cinder plugin will no longer use the deprecated "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" labels. It will use "topology.kubernetes.io/zone" and "topology.kubernetes.io/region" labels instead. (#99719, @jsafrane) [SIG Cloud Provider and Storage]
• OpenStack Cinder CSI migration is on by default, Clinder CSI driver must be installed on clusters on OpenStack for Cinder volumes to work. (#98538, @dims) [SIG Storage]
• Package pkg/kubelet/server migrated to structured logging Exit code changed from 255 to 1 (#99838, @adisky) [SIG Node]
• Pkg/kubelet/kuberuntime/kuberuntime_manager.go migrated to structured logging Exit code changed from 255 to 1 (#99841, @adisky) [SIG Instrumentation and Node]

Changes by Kind

Deprecation

• Kubeadm: the deprecated kube-dns is no longer supported as an option. If "ClusterConfiguration.dns.type" is set to "kube-dns" kubeadm will now throw an error. (#99646, @rajansandeep) [SIG Cluster Lifecycle]

• Remove deprecated --generator --replicas --service-generator --service-overrides --schedule from kubectl run Deprecate --serviceaccount --hostport --requests --limits in kubectl run (#99732, @soltysh) [SIG CLI and Testing]

• audit.k8s.io/v1beta1 and audit.k8s.io/v1alpha1 audit policy configuration and audit events are deprecated in favor of audit.k8s.io/v1, available since v1.13. kube-apiserver invocations that specify alpha or beta policy configurations with --audit-policy-file, or explicitly request alpha or beta audit events with --audit-log-version / --audit-webhook-version must update to use audit.k8s.io/v1 and accept audit.k8s.io/v1 events prior to v1.24. (#98858, @carlory) [SIG Auth]

• diskformat stroage class parameter for in-tree vSphere volume plugin is deprecated as of v1.21 release. Please consider updating storageclass and remove diskformat parameter. vSphere CSI Driver does not support diskformat storageclass parameter.

  vSphere releases less than 67u3 are deprecated as of v1.21. Please consider upgrading vSphere to 67u3 or above. vSphere CSI Driver requires minimum vSphere 67u3.

  VM Hardware version less than 15 is deprecated as of v1.21. Please consider upgrading the Node VM Hardware version to 15 or above. vSphere CSI Driver recommends Node VM's Hardware version set to at least vmx-15.

  Multi vCenter support is deprecated as of v1.21. If you have a Kubernetes cluster spanning across multiple vCenter servers, please consider moving all k8s nodes to a single vCenter Server. vSphere CSI Driver does not support Kubernetes deployment spanning across multiple vCenter servers.

  Support for these deprecations will be available till Kubernetes v1.24. (#98546, @divyenpatel) [SIG Cloud Provider and Storage]

API Change

• 1. PodAffinityTerm includes a namespaceSelector field to allow selecting eligible namespaces based on their labels.
  2. A new CrossNamespacePodAffinity quota scope API that allows restricting which namespaces allowed to use PodAffinityTerm with corss-namespace reference via namespaceSelector or namespaces fields. (#98582, @ahg-g) [SIG API Machinery, Apps, Auth and Testing]
• Add a default metadata name labels for selecting any namespace by its name. (#96968, @jayunit100) [SIG API Machinery, Apps, Cloud Provider, Storage and Testing]
• Added .spec.completionMode field to Job, with accepted values NonIndexed (default) and Indexed (#98441, @alculquicondor) [SIG Apps and CLI]
• Clarified NetworkPolicy policyTypes documentation (#97216, @joejulian) [SIG Network]
• DaemonSets accept a MaxSurge integer or percent on their rolling update strategy that will launch the updated pod on nodes and wait for those pods to go ready before marking the old out-of-date pods as deleted. This allows workloads to avoid downtime during upgrades when deployed using DaemonSets. This feature is alpha and is behind the DaemonSetUpdateSurge feature gate. (#96441, @smarterclayton) [SIG Apps and Testing]
• EndpointSlice API is now GA. The EndpointSlice topology field has been removed from the GA API and will be replaced by a new per Endpoint Zone field. If the topology field was previously used, it will be converted into an annotation in the v1 Resource. The discovery.k8s.io/v1alpha1 API is removed. (#99662, @swetharepakula) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network and Testing]
• EndpointSlice Controllers are now GA. The EndpointSlice Controller will not populate the deprecatedTopology field and will only provide topology information through the zone and nodeName fields. (#99870, @swetharepakula) [SIG API Machinery, Apps, Auth, Network and Testing]
• IngressClass resource can now reference a resource in a specific namespace for implementation-specific configuration(previously only Cluster-level resources were allowed). This feature can be enabled using the IngressClassNamespacedParams feature gate. (#99275, @hbagdi) [SIG API Machinery, CLI and Network]
• Introduce conditions for PodDisruptionBudget (#98127, @mortent) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Cluster Lifecycle and Instrumentation]
• Jobs API has a new .spec.suspend field that can be used to suspend and resume Jobs (#98727, @adtac) [SIG API Machinery, Apps, Node, Scheduling and Testing]
• Kubelet Graceful Node Shutdown feature is now beta. (#99735, @bobbypage) [SIG Node]
• Limit the quest value of hugepage to integer multiple of page size. (#98515, @lala123912) [SIG Apps]
• One new field "InternalTrafficPolicy" in Service is added. It specifies if the cluster internal traffic should be routed to all endpoints or node-local endpoints only. "Cluster" routes internal traffic to a Service to all endpoints. "Local" routes traffic to node-local endpoints only, and traffic is dropped if no node-local endpoints are ready. The default value is "Cluster". (#96600, @maplain) [SIG API Machinery, Apps and Network]
• PodSecurityPolicy only stores "generic" as allowed volume type if the GenericEphemeralVolume feature gate is enabled (#98918, @pohly) [SIG Auth and Security]
• Promote CronJobs to batch/v1 (#99423, @soltysh) [SIG API Machinery, Apps, CLI and Testing]
• Remove support for building Kubernetes with bazel. (#99561, @BenTheElder) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scalability, Scheduling, Storage, Testing and Windows]
• Setting loadBalancerClass in load balancer type of service is available with this PR. Users who want to use a custom load balancer can specify loadBalancerClass to achieve it. (#98277, @XudongLiuHarold) [SIG API Machinery, Apps, Cloud Provider and Network]
• Storage capacity tracking (= the CSIStorageCapacity feature) is beta, storage.k8s.io/v1alpha1/VolumeAttachment and storage.k8s.io/v1alpha1/CSIStorageCapacity objects are deprecated (#99641, @pohly) [SIG API Machinery, Apps, Auth, Scheduling, Storage and Testing]
• Support for Indexed Job: a Job that is considered completed when Pods associated to indexes from 0 to (.spec.completions-1) have succeeded. (#98812, @alculquicondor) [SIG Apps and CLI]
• The apiserver now resets managedFields that got corrupted by a mutating admission controller. (#98074, @kwiesmueller) [SIG API Machinery and Testing]
• controller.kubernetes.io/pod-deletion-cost annotation can be set to offer a hint on the cost of deleting a pod compared to other pods belonging to the same ReplicaSet. Pods with lower deletion cost are deleted first. This is an alpha feature. (#99163, @ahg-g) [SIG Apps]

Feature

• A client-go metric, rest_client_exec_plugin_call_total, has been added to track total calls to client-go credential plugins. (#98892, @ankeesler) [SIG API Machinery, Auth, Cluster Lifecycle and Instrumentation]
• Add --use-protocol-buffers flag to kubectl top pods and nodes (#96655, @serathius) [SIG CLI]
• Add support to generate client-side binaries for new darwin/arm64 platform (#97743, @dims) [SIG Release and Testing]
• Added ephemeral_volume_controller_create[_failures]_total counters to kube-controller-manager metrics (#99115, @pohly) [SIG API Machinery, Apps, Cluster Lifecycle, Instrumentation and Storage]
• Adds alpha feature VolumeCapacityPriority which makes the scheduler prioritize nodes based on the best matching size of statically provisioned PVs across multiple topologies. (#96347, @cofyc) [SIG Apps, Network, Scheduling, Storage and Testing]
• Adds two new metrics to cronjobs, a histogram to track the time difference when a job is created and the expected time when it should be created, and a gauge for the missed schedules of a cronjob (#99341, @alaypatel07) [SIG Apps and Instrumentation]
• Alpha implementation of Kubectl Command Headers: SIG CLI KEP 859 enabled when KUBECTL_COMMAND_HEADERS environment variable set on the client command line.
  • To enable: export KUBECTL_COMMAND_HEADERS=1; kubectl ... (#98952, @seans3) [SIG API Machinery and CLI]
• Component owner can configure the allowlist of metric label with flag '--allow-metric-labels'. (#99738, @YoyinZyc) [SIG API Machinery, Cluster Lifecycle and Instrumentation]
• Disruption controller only sends one event per PodDisruptionBudget if scale can't be computed (#98128, @mortent) [SIG Apps]
• EndpointSliceNodeName will always be enabled, so NodeName will always be available in the v1beta1 API. (#99746, @swetharepakula) [SIG Apps and Network]
• Graduate CRIContainerLogRotation feature gate to GA. (#99651, @umohnani8) [SIG Node and Testing]
• Kube-proxy iptables: new metric sync_proxy_rules_iptables_total that exposes the number of rules programmed per table in each iteration (#99653, @aojea) [SIG Instrumentation and Network]
• Kube-scheduler now logs plugin scoring summaries at --v=4 (#99411, @damemi) [SIG Scheduling]
• Kubeadm: a warning to user as ipv6 site-local is deprecated (#99574, @pacoxu) [SIG Cluster Lifecycle and Network]
• Kubeadm: apply the "node.kubernetes.io/exclude-from-external-load-balancers" label on control plane nodes during "init", "join" and "upgrade" to preserve backwards compatibility with the lagacy LB mode where nodes labeled as "master" where excluded. To opt-out you can remove the label from a node. See #97543 and the linked KEP for more details. (#98269, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: if the user has customized their image repository via the kubeadm configuration, pass the custom pause image repository and tag to the kubelet via --pod-infra-container-image not only for Docker but for all container runtimes. This flag tells the kubelet that it should not garbage collect the image. (#99476, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: promote IPv6DualStack feature gate to Beta (#99294, @pacoxu) [SIG Cluster Lifecycle]
• Kubectl version changed to write a warning message to stderr if the client and server version difference exceeds the supported version skew of +/-1 minor version. (#98250, @brianpursley) [SIG CLI]
• Kubernetes is now built with Golang 1.16 (#98572, @justaugustus) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Release and Testing]
• Persistent Volumes formatted with the btrfs filesystem will now automatically resize when expanded. (#99361, @Novex) [SIG Storage]
• Remove cAdvisor json metrics api collected by Kubelet (#99236, @pacoxu) [SIG Node]
• Sysctls is now GA and locked to default (#99158, @wgahnagl) [SIG Node]
• The NodeAffinity plugin implements the PreFilter extension, offering enhanced performance for Filter. (#99213, @AliceZhang2016) [SIG Scheduling]
• The endpointslice mirroring controller mirrors endpoints annotations and labels to the generated endpoint slices, it also ensures that updates on any of these fields are mirrored. The well-known annotation endpoints.kubernetes.io/last-change-trigger-time is skipped and not mirrored. (#98116, @aojea) [SIG Apps, Network and Testing]
• Update the latest validated version of Docker to 20.10 (#98977, @neolit123) [SIG CLI, Cluster Lifecycle and Node]
• Upgrade node local dns to 1.17.0 for better IPv6 support (#99749, @pacoxu) [SIG Cloud Provider and Network]
• Users might specify the kubectl.kubernetes.io/default-exec-container annotation in a Pod to preselect container for kubectl commands. (#99581, @mengjiao-liu) [SIG CLI]
• When downscaling ReplicaSets, ready and creation timestamps are compared in a logarithmic scale. (#99212, @damemi) [SIG Apps and Testing]
• When the kubelet is watching a ConfigMap or Secret purely in the context of setting environment variables for containers, only hold that watch for a defined duration before cancelling it. This change reduces the CPU and memory usage of the kube-apiserver in large clusters. (#99393, @chenyw1990) [SIG API Machinery, Node and Testing]
• WindowsEndpointSliceProxying feature gate has graduated to beta and is enabled by default. This means kube-proxy will read from EndpointSlices instead of Endpoints on Windows by default. (#99794, @robscott) [SIG Network]

Bug or Regression

• Creating a PVC with DataSource should fail for non-CSI plugins. (#97086, @xing-yang) [SIG Apps and Storage]
• EndpointSlice controller is now less likely to emit FailedToUpdateEndpointSlices events. (#99345, @robscott) [SIG Apps and Network]
• EndpointSliceMirroring controller is now less likely to emit FailedToUpdateEndpointSlices events. (#99756, @robscott) [SIG Apps and Network]
• Fix --ignore-errors does not take effect if multiple logs are printed and unfollowed (#97686, @wzshiming) [SIG CLI]
• Fix bug that would let the Horizontal Pod Autoscaler scale down despite at least one metric being unavailable/invalid (#99514, @mikkeloscar) [SIG Apps and Autoscaling]
• Fix cgroup handling for systemd with cgroup v2 (#98365, @odinuge) [SIG Node]
• Fix smb mount PermissionDenied issue on Windows (#99550, @andyzhangx) [SIG Cloud Provider, Storage and Windows]
• Fixed a bug that causes smaller number of conntrack-max being used under CPU static policy. (#99225, @xh4n3) (#99613, @xh4n3) [SIG Network]
• Fixed bug that caused cAdvisor to incorrectly detect single-socket multi-NUMA topology. (#99315, @iwankgb) [SIG Node]
• Fixes add-on manager leader election (#98968, @liggitt) [SIG Cloud Provider]
• Improved update time of pod statuses following new probe results. (#98376, @matthyx) [SIG Node and Testing]
• Kube-apiserver: an update of a pod with a generic ephemeral volume dropped that volume if the feature had been disabled since creating the pod with such a volume (#99446, @pohly) [SIG Apps, Node and Storage]
• Kubeadm: skip validating pod subnet against node-cidr-mask when allocate-node-cidrs is set to be false (#98984, @SataQiu) [SIG Cluster Lifecycle]
• On single-stack configured (IPv4 or IPv6, but not both) clusters, Services which are both headless (no clusterIP) and selectorless (empty or undefined selector) will report ipFamilyPolicy RequireDualStack and will have entries in ipFamilies[] for both IPv4 and IPv6. This is a change from alpha, but does not have any impact on the manually-specified Endpoints and EndpointSlices for the Service. (#99555, @thockin) [SIG Apps and Network]
• Resolves spurious Failed to list *v1.Secret or Failed to list *v1.ConfigMap messages in kubelet logs. (#99538, @liggitt) [SIG Auth and Node]
• Return zero time (midnight on Jan. 1, 1970) instead of negative number when reporting startedAt and finishedAt of the not started or a running Pod when using dockershim as a runtime. (#99585, @Iceber) [SIG Node]
• Stdin is now only passed to client-go exec credential plugins when it is detected to be an interactive terminal. Previously, it was passed to client-go exec plugins when *stdout- was detected to be an interactive terminal. (#99654, @ankeesler) [SIG API Machinery and Auth]
• The maximum number of ports allowed in EndpointSlices has been increased from 100 to 20,000 (#99795, @robscott) [SIG Network]
• Updates the commands
  • kubectl kustomize {arg}
  • kubectl apply -k {arg} to use same code as kustomize CLI v4.0.5
  • [v4.0.5]: https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv4.0.5 (#98946, @monopole) [SIG API Machinery, Architecture, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node and Storage]
• When a CNI plugin returns dual-stack pod IPs, kubelet will now try to respect the "primary IP family" of the cluster by picking a primary pod IP of the same family as the (primary) node IP, rather than assuming that the CNI plugin returned the IPs in the order the administrator wanted (since some CNI plugins don't allow configuring this). (#97979, @danwinship) [SIG Network and Node]
• When using Containerd on Windows, the "C:\Windows\System32\drivers\etc\hosts" file will now be managed by kubelet. (#83730, @claudiubelu) [SIG Node and Windows]
• VolumeBindingArgs now allow BindTimeoutSeconds to be set as zero, while the value zero indicates no waiting for the checking of volume binding operation. (#99835, @chendave) [SIG Scheduling and Storage]
• kubectl exec and kubectl attach now honor the --quiet flag which suppresses output from the local binary that could be confused by a script with the remote command output (all non-failure output is hidden). In addition, print inline with exec and attach the list of alternate containers when we default to the first spec.container. (#99004, @smarterclayton) [SIG CLI]

Other (Cleanup or Flake)

• Apiserver_request_duration_seconds is promoted to stable status. (#99925, @logicalhan) [SIG API Machinery, Instrumentation and Testing]
• Apiserver_request_total is promoted to stable status and no longer has a content-type dimensions, so any alerts/charts which presume the existence of this will fail. This is however, unlikely to be the case since it was effectively an unbounded dimension in the first place. (#99788, @logicalhan) [SIG API Machinery, Instrumentation and Testing]
• EndpointSlice generation is now incremented when labels change. (#99750, @robscott) [SIG Network]
• Featuregate AllowInsecureBackendProxy is promoted to GA (#99658, @deads2k) [SIG API Machinery]
• Migrate pkg/kubelet/(eviction) to structured logging (#99032, @yangjunmyfm192085) [SIG Node]
• Migrate deployment controller log messages to structured logging (#97507, @aldudko) [SIG Apps]
• Migrate pkg/kubelet/cloudresource to structured logging (#98999, @sladyn98) [SIG Node]
• Migrate pkg/kubelet/cri/remote logs to structured logging (#98589, @chenyw1990) [SIG Node]
• Migrate pkg/kubelet/kuberuntime/kuberuntime_container.go logs to structured logging (#96973, @chenyw1990) [SIG Instrumentation and Node]
• Migrate pkg/kubelet/status to structured logging (#99836, @navidshaikh) [SIG Instrumentation and Node]
• Migrate pkg/kubelet/token to structured logging (#99264, @palnabarun) [SIG Auth, Instrumentation and Node]
• Migrate pkg/kubelet/util to structured logging (#99823, @navidshaikh) [SIG Instrumentation and Node]
• Migrate proxy/userspace/proxier.go logs to structured logging (#97837, @JornShen) [SIG Network]
• Migrate some kubelet/metrics log messages to structured logging (#98627, @jialaijun) [SIG Instrumentation and Node]
• Process start time on Windows now uses current process information (#97491, @jsturtevant) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Windows]

Uncategorized

• Migrate pkg/kubelet/stats to structured logging (#99607, @krzysiekg) [SIG Node]
• The DownwardAPIHugePages feature is beta. Users may use the feature if all workers in their cluster are min 1.20 version. The feature will be enabled by default in all installations in 1.22. (#99610, @derekwaynecarr) [SIG Node]

Dependencies

Added

• github.com/go-errors/errors: v1.0.1
• github.com/gobuffalo/here: v0.6.0
• github.com/google/shlex: e7afc7f
• github.com/markbates/pkger: v0.17.1
• github.com/monochromegane/go-gitignore: 205db1a
• github.com/niemeyer/pretty: a10e7ca
• github.com/xlab/treeprint: a009c39
• go.starlark.net: 8dd3e2e
• golang.org/x/term: 6a3ed07
• sigs.k8s.io/kustomize/api: v0.8.5
• sigs.k8s.io/kustomize/cmd/config: v0.9.7
• sigs.k8s.io/kustomize/kustomize/v4: v4.0.5
• sigs.k8s.io/kustomize/kyaml: v0.10.15

Changed

• dmitri.shuralyov.com/gpu/mtl: 666a987 → 28db891
• github.com/creack/pty: v1.1.7 → v1.1.9
• github.com/go-openapi/spec: v0.19.3 → v0.19.5
• github.com/go-openapi/strfmt: v0.19.3 → v0.19.5
• github.com/go-openapi/validate: v0.19.5 → v0.19.8
• github.com/google/cadvisor: v0.38.7 → v0.38.8
• github.com/kr/text: v0.1.0 → v0.2.0
• github.com/mattn/go-runewidth: v0.0.2 → v0.0.7
• github.com/olekukonko/tablewriter: a0225b3 → v0.0.4
• github.com/sergi/go-diff: v1.0.0 → v1.1.0
• golang.org/x/crypto: 7f63de1 → 5ea612d
• golang.org/x/exp: 6cc2880 → 85be41e
• golang.org/x/mobile: d2bd2a2 → e6ae53a
• golang.org/x/mod: v0.3.0 → ce943fd
• golang.org/x/net: 69a7880 → 3d97a24
• golang.org/x/sys: 5cba982 → a50acf3
• golang.org/x/time: 3af7569 → f8bda1e
• golang.org/x/tools: 113979e → v0.1.0
• gopkg.in/check.v1: 41f04d3 → 8fa4692
• gopkg.in/yaml.v2: v2.2.8 → v2.4.0
• k8s.io/kube-openapi: d219536 → 591a79e
• k8s.io/system-validators: v1.3.0 → v1.4.0

Removed

• github.com/codegangsta/negroni: v1.0.0
• github.com/golangplus/bytes: 45c989f
• github.com/golangplus/fmt: 2a5d6d7
• github.com/gorilla/context: v1.1.1
• github.com/kr/pty: v1.1.5
• sigs.k8s.io/kustomize: v2.0.3+incompatible

v1.21.0-beta.0

Downloads for v1.21.0-beta.0

Source Code

Client binaries

Server binaries

Node binaries

Changelog since v1.21.0-alpha.3

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• The metric storage_operation_errors_total is not removed, but is marked deprecated, and the metric storage_operation_status_count is marked deprecated. In both cases the storage_operation_duration_seconds metric can be used to recover equivalent counts (using status=fail-unknown in the case of storage_operations_errors_total). (#99045, @mattcary) [SIG Instrumentation and Storage]

Changes by Kind

Deprecation

• The batch/v2alpha1 CronJob type definitions and clients are deprecated and removed. (#96987, @soltysh) [SIG API Machinery, Apps, CLI and Testing]

API Change

• Cluster admins can now turn off /debug/pprof and /debug/flags/v endpoint in kubelet by setting enableProfilingHandler and enableDebugFlagsHandler to false in their kubelet configuration file. enableProfilingHandler and enableDebugFlagsHandler can be set to true only when enableDebuggingHandlers is also set to true. (#98458, @SaranBalaji90) [SIG Node]
• The BoundServiceAccountTokenVolume feature has been promoted to beta, and enabled by default.
  • This changes the tokens provided to containers at /var/run/secrets/kubernetes.io/serviceaccount/token to be time-limited, auto-refreshed, and invalidated when the containing pod is deleted.
  • Clients should reload the token from disk periodically (once per minute is recommended) to ensure they continue to use a valid token. k8s.io/client-go version v11.0.0+ and v0.15.0+ reload tokens automatically.
  • By default, injected tokens are given an extended lifetime so they remain valid even after a new refreshed token is provided. The metric serviceaccount_stale_tokens_total can be used to monitor for workloads that are depending on the extended lifetime and are continuing to use tokens even after a refreshed token is provided to the container. If that metric indicates no existing workloads are depending on extended lifetimes, injected token lifetime can be shortened to 1 hour by starting kube-apiserver with --service-account-extend-token-expiration=false. (#95667, @zshihang) [SIG API Machinery, Auth, Cluster Lifecycle and Testing]

Feature

• A new histogram metric to track the time it took to delete a job by the ttl-after-finished controller (#98676, @ahg-g) [SIG Apps and Instrumentation]
• AWS cloudprovider supports auto-discovering subnets without any kubernetes.io/cluster/ tags. It also supports additional service annotation service.beta.kubernetes.io/aws-load-balancer-subnets to manually configure the subnets. (#97431, @kishorj) [SIG Cloud Provider]
• Add --permit-address-sharing flag to kube-apiserver to listen with SO_REUSEADDR. While allowing to listen on wildcard IPs like 0.0.0.0 and specific IPs in parallel, it avoid waiting for the kernel to release socket in TIME_WAIT state, and hence, considably reducing kube-apiserver restart times under certain conditions. (#93861, @sttts) [SIG API Machinery]
• Add csi_operations_seconds metric on kubelet that exposes CSI operations duration and status for node CSI operations. (#98979, @Jiawei0227) [SIG Instrumentation and Storage]
• Add migrated field into storage_operation_duration_seconds metric (#99050, @Jiawei0227) [SIG Apps, Instrumentation and Storage]
• Add bash-completion for comma separated list on kubectl get (#98301, @phil9909) [SIG CLI]
• Added support for installing arm64 node artifacts. (#99242, @liu-cong) [SIG Cloud Provider]
• Feature gate RootCAConfigMap is graduated to GA in 1.21 and will be removed in 1.22. (#98033, @zshihang) [SIG API Machinery and Auth]
• Kubeadm: during "init" and "join" perform preflight validation on the host / node name and throw warnings if a name is not compliant (#99194, @pacoxu) [SIG Cluster Lifecycle]
• Kubectl: kubectl get will omit managed fields by default now. Users could set --show-managed-fields to true to show managedFields when the output format is either json or yaml. (#96878, @knight42) [SIG CLI and Testing]
• Metrics can now be disabled explicitly via a command line flag (i.e. '--disabled-metrics=bad_metric1,bad_metric2') (#99217, @logicalhan) [SIG API Machinery, Cluster Lifecycle and Instrumentation]
• TTLAfterFinished is now beta and enabled by default (#98678, @ahg-g) [SIG Apps and Auth]
• The RunAsGroup feature has been promoted to GA in this release. (#94641, @krmayankk) [SIG Auth and Node]
• Turn CronJobControllerV2 on by default. (#98878, @soltysh) [SIG Apps]
• UDP protocol support for Agnhost connect subcommand (#98639, @knabben) [SIG Testing]
• Upgrades IPv6Dualstack to Beta and turns it on by default. Clusters new and existing will not be affected until user starting adding secondary pod and service cidrs cli flags as described here: https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/563-dual-stack (#98969, @khenidak) [SIG API Machinery, Apps, Cloud Provider, Network and Node]

Documentation

• Fix ALPHA stability level reference link (#98641, @Jeffwan) [SIG Auth, Cloud Provider, Instrumentation and Storage]

Failing Test

• Escape the special characters like [, ] and that exist in vsphere windows path (#98830, @liyanhui1228) [SIG Storage and Windows]
• Kube-proxy: fix a bug on UDP NodePort Services where stale conntrack entries may blackhole the traffic directed to the NodePort. (#98305, @aojea) [SIG Network]

Bug or Regression

• Add missing --kube-api-content-type in kubemark hollow template (#98911, @Jeffwan) [SIG Scalability and Testing]
• Avoid duplicate error messages when runing kubectl edit quota (#98201, @pacoxu) [SIG API Machinery and Apps]
• Cleanup subnet in frontend IP configs to prevent huge subnet request bodies in some scenarios. (#98133, @nilo19) [SIG Cloud Provider]
• Fix errors when accessing Windows container stats for Dockershim (#98510, @jsturtevant) [SIG Node and Windows]
• Fixes spurious errors about IPv6 in kube-proxy logs on nodes with IPv6 disabled. (#99127, @danwinship) [SIG Network and Node]
• In the method that ensures that the docker and containerd are in the correct containers with the proper OOM score set up, fixed the bug of identifying containerd process. (#97888, @pacoxu) [SIG Node]
• Kubelet now cleans up orphaned volume directories automatically (#95301, @lorenz) [SIG Node and Storage]
• When dynamically provisioning Azure File volumes for a premium account, the requested size will be set to 100GB if the request is initially lower than this value to accommodate Azure File requirements. (#99122, @huffmanca) [SIG Cloud Provider and Storage]

Other (Cleanup or Flake)

• APIs for kubelet annotations and labels from k8s.io/kubernetes/pkg/kubelet/apis are now available under k8s.io/kubelet/pkg/apis/ (#98931, @michaelbeaumont) [SIG Apps, Auth and Node]
• Migrate pkg/kubelet/(pod, pleg) to structured logging (#98990, @gjkim42) [SIG Instrumentation and Node]
• Migrate pkg/kubelet/nodestatus to structured logging (#99001, @QiWang19) [SIG Node]
• Migrate pkg/kubelet/server logs to structured logging (#98643, @chenyw1990) [SIG Node]
• Migrate proxy/winkernel/proxier.go logs to structured logging (#98001, @JornShen) [SIG Network and Windows]
• Migrate scheduling_queue.go to structured logging (#98358, @tanjing2020) [SIG Scheduling]
• Several flags related to the deprecated dockershim which are present in the kubelet command line are now deprecated. (#98730, @dims) [SIG Node]
• The deprecated feature gates CSIDriverRegistry, BlockVolume and CSIBlockVolume are now unconditionally enabled and can no longer be specified in component invocations. (#98021, @gavinfish) [SIG Storage]

Dependencies

Added

Nothing has changed.

Changed

• sigs.k8s.io/structured-merge-diff/v4: v4.0.2 → v4.0.3

Removed

Nothing has changed.

v1.21.0-alpha.3

Downloads for v1.21.0-alpha.3

Source Code

Client binaries

Server binaries

Node binaries

Changelog since v1.21.0-alpha.2

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Newly provisioned PVs by gce-pd will no longer have the beta FailureDomain label. gce-pd volume plugin will start to have GA topology label instead. (#98700, @Jiawei0227) [SIG Cloud Provider, Storage and Testing]
• Remove alpha CSIMigrationXXComplete flag and add alpha InTreePluginXXUnregister flag. Deprecate CSIMigrationvSphereComplete flag and it will be removed in 1.22. (#98243, @Jiawei0227) [SIG Node and Storage]

Changes by Kind

API Change

• Adds support for portRange / EndPort in Network Policy (#97058, @rikatz) [SIG Apps and Network]
• Fixes using server-side apply with APIService resources (#98576, @kevindelgado) [SIG API Machinery, Apps and Testing]
• Kubernetes is now built using go1.15.7 (#98363, @cpanato) [SIG Cloud Provider, Instrumentation, Node, Release and Testing]
• Scheduler extender filter interface now can report unresolvable failed nodes in the new field FailedAndUnresolvableNodes of ExtenderFilterResult struct. Nodes in this map will be skipped in the preemption phase. (#92866, @cofyc) [SIG Scheduling]

Feature

• A lease can only attach up to 10k objects. (#98257, @lingsamuel) [SIG API Machinery]
• Add ignore-errors flag for drain, support none-break drain in group (#98203, @yuzhiquan) [SIG CLI]
• Base-images: Update to debian-iptables:buster-v1.4.0
  • Uses iptables 1.8.5
  • base-images: Update to debian-base:buster-v1.3.0
  • cluster/images/etcd: Build etcd:3.4.13-2 image
    • Uses debian-base:buster-v1.3.0 (#98401, @pacoxu) [SIG Testing]
• Export NewDebuggingRoundTripper function and DebugLevel options in the k8s.io/client-go/transport package. (#98324, @atosatto) [SIG API Machinery]
• Kubectl wait ensures that observedGeneration >= generation if applicable (#97408, @KnicKnic) [SIG CLI]
• Kubernetes is now built using go1.15.8 (#98834, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]
• New admission controller "denyserviceexternalips" is available. Clusters which do not *need- the Service "externalIPs" feature should enable this controller and be more secure. (#97395, @thockin) [SIG API Machinery]
• Overall, enable the feature of PreferNominatedNode will improve the performance of scheduling where preemption might frequently happen, but in theory, enable the feature of PreferNominatedNode, the pod might not be scheduled to the best candidate node in the cluster. (#93179, @chendave) [SIG Scheduling and Testing]
• Pause image upgraded to 3.4.1 in kubelet and kubeadm for both Linux and Windows. (#98205, @pacoxu) [SIG CLI, Cloud Provider, Cluster Lifecycle, Node, Testing and Windows]
• The ServiceAccountIssuerDiscovery feature has graduated to GA, and is unconditionally enabled. The ServiceAccountIssuerDiscovery feature-gate will be removed in 1.22. (#98553, @mtaufen) [SIG API Machinery, Auth and Testing]

Documentation

• Feat: azure file migration go beta in 1.21. Feature gates CSIMigration to Beta (on by default) and CSIMigrationAzureFile to Beta (off by default since it requires installation of the AzureFile CSI Driver) The in-tree AzureFile plugin "kubernetes.io/azure-file" is now deprecated and will be removed in 1.23. Users should enable CSIMigration + CSIMigrationAzureFile features and install the AzureFile CSI Driver (https://github.com/kubernetes-sigs/azurefile-csi-driver) to avoid disruption to existing Pod and PVC objects at that time. Users should start using the AzureFile CSI Driver directly for any new volumes. (#96293, @andyzhangx) [SIG Cloud Provider]

Failing Test

• Kubelet: the HostPort implementation in dockershim was not taking into consideration the HostIP field, causing that the same HostPort can not be used with different IP addresses. This bug causes the conformance test "HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol" to fail. (#98755, @aojea) [SIG Cloud Provider, Network and Node]

Bug or Regression

• Fix NPE in ephemeral storage eviction (#98261, @wzshiming) [SIG Node]
• Fixed a bug that on k8s nodes, when the policy of INPUT chain in filter table is not ACCEPT, healthcheck nodeport would not work. Added iptables rules to allow healthcheck nodeport traffic. (#97824, @hanlins) [SIG Network]
• Fixed kube-proxy container image architecture for non amd64 images. (#98526, @saschagrunert) [SIG API Machinery, Release and Testing]
• Fixed provisioning of Cinder volumes migrated to CSI when StorageClass with AllowedTopologies was used. (#98311, @jsafrane) [SIG Storage]
• Fixes a panic in the disruption budget controller for PDB objects with invalid selectors (#98750, @mortent) [SIG Apps]
• Fixes connection errors when using --volume-host-cidr-denylist or --volume-host-allow-local-loopback (#98436, @liggitt) [SIG Network and Storage]
• If the user specifies an invalid timeout in the request URL, the request will be aborted with an HTTP 400.
  • in cases where the client specifies a timeout in the request URL, the overall request deadline is shortened now since the deadline is setup as soon as the request is received by the apiserver. (#96901, @tkashem) [SIG API Machinery and Testing]
• Kubeadm: Some text in the kubeadm upgrade plan output has changed. If you have scripts or other automation that parses this output, please review these changes and update your scripts to account for the new output. (#98728, @stmcginnis) [SIG Cluster Lifecycle]
• Kubeadm: fix a bug where external credentials in an existing admin.conf prevented the CA certificate to be written in the cluster-info ConfigMap. (#98882, @kvaps) [SIG Cluster Lifecycle]
• Kubeadm: fix bad token placeholder text in "config print *-defaults --help" (#98839, @Mattias-) [SIG Cluster Lifecycle]
• Kubeadm: get k8s CI version markers from k8s infra bucket (#98836, @hasheddan) [SIG Cluster Lifecycle and Release]
• Mitigate CVE-2020-8555 for kube-up using GCE by preventing local loopback folume hosts. (#97934, @mattcary) [SIG Cloud Provider and Storage]
• Remove CSI topology from migrated in-tree gcepd volume. (#97823, @Jiawei0227) [SIG Cloud Provider and Storage]
• Sync node status during kubelet node shutdown. Adds an pod admission handler that rejects new pods when the node is in progress of shutting down. (#98005, @wzshiming) [SIG Node]
• Truncates a message if it hits the NoteLengthLimit when the scheduler records an event for the pod that indicates the pod has failed to schedule. (#98715, @carlory) [SIG Scheduling]
• We will no longer automatically delete all data when a failure is detected during creation of the volume data file on a CSI volume. Now we will only remove the data file and volume path. (#96021, @huffmanca) [SIG Storage]

Other (Cleanup or Flake)

• Fix the description of command line flags that can override --config (#98254, @changshuchao) [SIG Scheduling]
• Migrate scheduler/taint_manager.go structured logging (#98259, @tanjing2020) [SIG Apps]
• Migrate staging/src/k8s.io/apiserver/pkg/admission logs to structured logging (#98138, @lala123912) [SIG API Machinery]
• Resolves flakes in the Ingress conformance tests due to conflicts with controllers updating the Ingress object (#98430, @liggitt) [SIG Network and Testing]
• The default delegating authorization options now allow unauthenticated access to healthz, readyz, and livez. A system:masters user connecting to an authz delegator will not perform an authz check. (#98325, @deads2k) [SIG API Machinery, Auth, Cloud Provider and Scheduling]
• The e2e suite can be instructed not to wait for pods in kube-system to be ready or for all nodes to be ready by passing --allowed-not-ready-nodes=-1 when invoking the e2e.test program. This allows callers to run subsets of the e2e suite in scenarios other than perfectly healthy clusters. (#98781, @smarterclayton) [SIG Testing]
• The feature gates WindowsGMSA and WindowsRunAsUserName that are GA since v1.18 are now removed. (#96531, @ialidzhikov) [SIG Node and Windows]
• The new -gce-zones flag on the e2e.test binary instructs tests that check for information about how the cluster interacts with the cloud to limit their queries to the provided zone list. If not specified, the current behavior of asking the cloud provider for all available zones in multi zone clusters is preserved. (#98787, @smarterclayton) [SIG API Machinery, Cluster Lifecycle and Testing]

Dependencies

Added

• github.com/moby/spdystream: v0.2.0

Changed

• github.com/NYTimes/gziphandler: 56545f4 → v1.1.1
• github.com/container-storage-interface/spec: v1.2.0 → v1.3.0
• github.com/go-logr/logr: v0.2.0 → v0.4.0
• github.com/gogo/protobuf: v1.3.1 → v1.3.2
• github.com/kisielk/errcheck: v1.2.0 → v1.5.0
• github.com/yuin/goldmark: v1.1.27 → v1.2.1
• golang.org/x/sync: cd5d95a → 67f06af
• golang.org/x/tools: c1934b7 → 113979e
• k8s.io/klog/v2: v2.4.0 → v2.5.0
• sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.14 → v0.0.15

Removed

• github.com/docker/spdystream: 449fdfc

v1.21.0-alpha.2

Downloads for v1.21.0-alpha.2

Source Code

Client binaries

Server binaries

Node binaries

Changelog since v1.21.0-alpha.1

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Remove storage metrics storage_operation_errors_total, since we already have storage_operation_status_count.And add new field status for storage_operation_duration_seconds, so that we can know about all status storage operation latency. (#98332, @JornShen) [SIG Instrumentation and Storage]

Changes by Kind

Deprecation

• Remove the TokenRequest and TokenRequestProjection feature gates (#97148, @wawa0210) [SIG Node]
• Removing experimental windows container hyper-v support with Docker (#97141, @wawa0210) [SIG Node and Windows]
• The export query parameter (inconsistently supported by API resources and deprecated in v1.14) is fully removed. Requests setting this query parameter will now receive a 400 status response. (#98312, @deads2k) [SIG API Machinery, Auth and Testing]

API Change

• Enable SPDY pings to keep connections alive, so that kubectl exec and kubectl port-forward won't be interrupted. (#97083, @knight42) [SIG API Machinery and CLI]

Documentation

• Official support to build kubernetes with docker-machine / remote docker is removed. This change does not affect building kubernetes with docker locally. (#97935, @adeniyistephen) [SIG Release and Testing]
• Set kubelet option --volume-stats-agg-period to negative value to disable volume calculations. (#96675, @pacoxu) [SIG Node]

Bug or Regression

• Clean ReplicaSet by revision instead of creation timestamp in deployment controller (#97407, @waynepeking348) [SIG Apps]
• Ensure that client-go's EventBroadcaster is safe (non-racy) during shutdown. (#95664, @DirectXMan12) [SIG API Machinery]
• Fix azure file migration issue (#97877, @andyzhangx) [SIG Auth, Cloud Provider and Storage]
• Fix kubelet from panic after getting the wrong signal (#98200, @wzshiming) [SIG Node]
• Fix repeatedly acquire the inhibit lock (#98088, @wzshiming) [SIG Node]
• Fixed a bug that the kubelet cannot start on BtrfS. (#98042, @gjkim42) [SIG Node]
• Fixed an issue with garbage collection failing to clean up namespaced children of an object also referenced incorrectly by cluster-scoped children (#98068, @liggitt) [SIG API Machinery and Apps]
• Fixed no effect namespace when exposing deployment with --dry-run=client. (#97492, @masap) [SIG CLI]
• Fixing a bug where a failed node may not have the NoExecute taint set correctly (#96876, @howieyuen) [SIG Apps and Node]
• Indentation of Resource Quota block in kubectl describe namespaces output gets correct. (#97946, @dty1er) [SIG CLI]
• KUBECTL_EXTERNAL_DIFF now accepts equal sign for additional parameters. (#98158, @dougsland) [SIG CLI]
• Kubeadm: fix a bug where "kubeadm join" would not properly handle missing names for existing etcd members. (#97372, @ihgann) [SIG Cluster Lifecycle]
• Kubelet should ignore cgroup driver check on Windows node. (#97764, @pacoxu) [SIG Node and Windows]
• Make podTopologyHints protected by lock (#95111, @choury) [SIG Node]
• Readjust kubelet_containers_per_pod_count bucket (#98169, @wawa0210) [SIG Instrumentation and Node]
• Scores from InterPodAffinity have stronger differentiation. (#98096, @leileiwan) [SIG Scheduling]
• Specifying the KUBE_TEST_REPO environment variable when e2e tests are executed will instruct the test infrastructure to load that image from a location within the specified repo, using a predefined pattern. (#93510, @smarterclayton) [SIG Testing]
• Static pods will be deleted gracefully. (#98103, @gjkim42) [SIG Node]
• Use network.Interface.VirtualMachine.ID to get the binded VM Skip standalone VM when reconciling LoadBalancer (#97635, @nilo19) [SIG Cloud Provider]

Other (Cleanup or Flake)

• Kubeadm: change the default image repository for CI images from 'gcr.io/kubernetes-ci-images' to 'gcr.io/k8s-staging-ci-images' (#97087, @SataQiu) [SIG Cluster Lifecycle]
• Migrate generic_scheduler.go and types.go to structured logging. (#98134, @tanjing2020) [SIG Scheduling]
• Migrate proxy/winuserspace/proxier.go logs to structured logging (#97941, @JornShen) [SIG Network]
• Migrate staging/src/k8s.io/apiserver/pkg/audit/policy/reader.go logs to structured logging. (#98252, @lala123912) [SIG API Machinery and Auth]
• Migrate staging\src\k8s.io\apiserver\pkg\endpoints logs to structured logging (#98093, @lala123912) [SIG API Machinery]
• Node (#96552, @pandaamanda) [SIG Apps, Cloud Provider, Node and Scheduling]
• The kubectl alpha debug command was scheduled to be removed in v1.21. (#98111, @pandaamanda) [SIG CLI]
• Update cri-tools to v1.20.0 (#97967, @rajibmitra) [SIG Cloud Provider]
• Windows nodes on GCE will take longer to start due to dependencies installed at node creation time. (#98284, @pjh) [SIG Cloud Provider]

Dependencies

Added

Nothing has changed.

Changed

• github.com/google/cadvisor: v0.38.6 → v0.38.7
• k8s.io/gengo: 83324d8 → b6c5ce2

Removed

Nothing has changed.

v1.21.0-alpha.1

Downloads for v1.21.0-alpha.1

Source Code

Client binaries

Server binaries

Node binaries

Changelog since v1.20.0

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Kube-proxy's IPVS proxy mode no longer sets the net.ipv4.conf.all.route_localnet sysctl parameter. Nodes upgrading will have net.ipv4.conf.all.route_localnet set to 1 but new nodes will inherit the system default (usually 0). If you relied on any behavior requiring net.ipv4.conf.all.route_localnet, you must set ensure it is enabled as kube-proxy will no longer set it automatically. This change helps to further mitigate CVE-2020-8558. (#92938, @lbernail) [SIG Network and Release]

Changes by Kind

Deprecation

• Deprecate the topologyKeys field in Service. This capability will be replaced with upcoming work around Topology Aware Subsetting and Service Internal Traffic Policy. (#96736, @andrewsykim) [SIG Apps]
• Kubeadm: deprecated command "alpha selfhosting pivot" is removed now. (#97627, @knight42) [SIG Cluster Lifecycle]
• Kubeadm: graduate the command kubeadm alpha kubeconfig user to kubeadm kubeconfig user. The kubeadm alpha kubeconfig user command is deprecated now. (#97583, @knight42) [SIG Cluster Lifecycle]
• Kubeadm: the "kubeadm alpha certs" command is removed now, please use "kubeadm certs" instead. (#97706, @knight42) [SIG Cluster Lifecycle]
• Remove the deprecated metrics "scheduling_algorithm_preemption_evaluation_seconds" and "binding_duration_seconds", suggest to use "scheduler_framework_extension_point_duration_seconds" instead. (#96447, @chendave) [SIG Cluster Lifecycle, Instrumentation, Scheduling and Testing]
• The PodSecurityPolicy API is deprecated in 1.21, and will no longer be served starting in 1.25. (#97171, @deads2k) [SIG Auth and CLI]

API Change

• Change the APIVersion proto name of BoundObjectRef from aPIVersion to apiVersion. (#97379, @kebe7jun) [SIG Auth]
• Promote Immutable Secrets/ConfigMaps feature to Stable. This allows to set Immutable field in Secrets or ConfigMap object to mark their contents as immutable. (#97615, @wojtek-t) [SIG Apps, Architecture, Node and Testing]

Feature

• Add flag --lease-max-object-size and metric etcd_lease_object_counts for kube-apiserver to config and observe max objects attached to a single etcd lease. (#97480, @lingsamuel) [SIG API Machinery, Instrumentation and Scalability]
• Add flag --lease-reuse-duration-seconds for kube-apiserver to config etcd lease reuse duration. (#97009, @lingsamuel) [SIG API Machinery and Scalability]
• Adds the ability to pass --strict-transport-security-directives to the kube-apiserver to set the HSTS header appropriately. Be sure you understand the consequences to browsers before setting this field. (#96502, @249043822) [SIG Auth]
• Kubeadm now includes CoreDNS v1.8.0. (#96429, @rajansandeep) [SIG Cluster Lifecycle]
• Kubeadm: add support for certificate chain validation. When using kubeadm in external CA mode, this allows an intermediate CA to be used to sign the certificates. The intermediate CA certificate must be appended to each signed certificate for this to work correctly. (#97266, @robbiemcmichael) [SIG Cluster Lifecycle]
• Kubeadm: amend the node kernel validation to treat CGROUP_PIDS, FAIR_GROUP_SCHED as required and CFS_BANDWIDTH, CGROUP_HUGETLB as optional (#96378, @neolit123) [SIG Cluster Lifecycle and Node]
• The Kubernetes pause image manifest list now contains an image for Windows Server 20H2. (#97322, @claudiubelu) [SIG Windows]
• The apimachinery util/net function used to detect the bind address ResolveBindAddress() takes into consideration global ip addresses on loopback interfaces when: - the host has default routes - there are no global IPs on those interfaces. in order to support more complex network scenarios like BGP Unnumbered RFC 5549 (#95790, @aojea) [SIG Network]

Bug or Regression

• Changelog

  General

  • Fix priority expander falling back to a random choice even though there is a higher priority option to choose
  • Clone kubernetes/kubernetes in update-vendor.sh shallowly, instead of fetching all revisions
  • Speed up binpacking by reducing the number of PreFilter calls (call once per pod instead of #pods*#nodes times)
  • Speed up finding unneeded nodes by 5x+ in very large clusters by reducing the number of PreFilter calls
  • Expose --max-nodes-total as a metric
  • Errors in IncreaseSize changed from type apiError to cloudProviderError
  • Make build-in-docker and test-in-docker work on Linux systems with SELinux enabled
  • Fix an error where existing nodes were not considered as destinations while finding place for pods in scale-down simulations
  • Remove redundant log lines and reduce severity around parsing kubeEnv
  • Don't treat nodes created by virtual kubelet as nodes from non-autoscaled node groups
  • Remove redundant logging around calculating node utilization
  • Add configurable --network and --rm flags for docker in Makefile
  • Subtract DaemonSet pods' requests from node allocatable in the denominator while computing node utilization
  • Include taints by condition when determining if a node is unready/still starting
  • Fix update-vendor.sh to work on OSX and zsh
  • Add best-effort eviction for DaemonSet pods while scaling down non-empty nodes
  • Add build support for ARM64

  AliCloud

  • Add missing daemonsets and replicasets to ALI example cluster role

  Apache CloudStack

  • Add support for Apache CloudStack

  AWS

  • Regenerate list of EC2 instances
  • Fix pricing endpoint in AWS China Region

  Azure

  • Add optional jitter on initial VMSS VM cache refresh, keep the refreshes spread over time
  • Serve from cache for the whole period of ongoing throttling
  • Fix unwanted VMSS VMs cache invalidations
  • Enforce setting the number of retries if cloud provider backoff is enabled
  • Don't update capacity if VMSS provisioning state is updating
  • Support allocatable resources overrides via VMSS tags
  • Add missing stable labels in template nodes
  • Proactively set instance status to deleting on node deletions

  Cluster API

  • Migrate interaction with the API from using internal types to using Unstructured
  • Improve tests to work better with constrained resources
  • Add support for node autodiscovery
  • Add support for --cloud-config
  • Update group identifier to use for Cluster API annotations

  Exoscale

  • Add support for Exoscale

  GCE

  • Decrease the number of GCE Read Requests made while deleting nodes
  • Base pricing of custom instances on their instance family type
  • Add pricing information for missing machine types
  • Add pricing information for different GPU types
  • Ignore the new topology.gke.io/zone label when comparing groups
  • Add missing stable labels to template nodes

  HuaweiCloud

  • Add auto scaling group support
  • Implement node group by AS
  • Implement getting desired instance number of node group
  • Implement increasing node group size
  • Implement TemplateNodeInfo
  • Implement caching instances

  IONOS

  • Add support for IONOS

  Kubemark

  • Skip non-kubemark nodes while computing node infos for node groups.

  Magnum

  • Add Magnum support in the Cluster Autoscaler helm chart

  Packet

  • Allow empty nodepools
  • Add support for multiple nodepools
  • Add pricing support

  Image

  Image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.20.0 (#97011, @towca) [SIG Cloud Provider]

• AcceleratorStats will be available in the Summary API of kubelet when cri_stats_provider is used. (#96873, @ruiwen-zhao) [SIG Node]

• Add limited lines to log when having tail option (#93920, @zhouya0) [SIG Node]

• Avoid systemd-logind loading configuration warning (#97950, @wzshiming) [SIG Node]

• Cloud-controller-manager: routes controller should not depend on --allocate-node-cidrs (#97029, @andrewsykim) [SIG Cloud Provider and Testing]

• Copy annotations with empty value when deployment rolls back (#94858, @waynepeking348) [SIG Apps]

• Detach volumes from vSphere nodes not tracked by attach-detach controller (#96689, @gnufied) [SIG Cloud Provider and Storage]

• Fix kubectl label error when local=true is set. (#97440, @pandaamanda) [SIG CLI]

• Fix Azure file share not deleted issue when the namespace is deleted (#97417, @andyzhangx) [SIG Cloud Provider and Storage]

• Fix CVE-2020-8555 for Gluster client connections. (#97922, @liggitt) [SIG Storage]

• Fix counting error in service/nodeport/loadbalancer quota check (#97451, @pacoxu) [SIG API Machinery, Network and Testing]

• Fix kubectl-convert import known versions (#97754, @wzshiming) [SIG CLI and Testing]

• Fix missing cadvisor machine metrics. (#97006, @lingsamuel) [SIG Node]

• Fix nil VMSS name when setting service to auto mode (#97366, @nilo19) [SIG Cloud Provider]

• Fix the panic when kubelet registers if a node object already exists with no Status.Capacity or Status.Allocatable (#95269, @SataQiu) [SIG Node]

• Fix the regression with the slow pods termination. Before this fix pods may take an additional time to terminate - up to one minute. Reversing the change that ensured that CNI resources cleaned up when the pod is removed on API server. (#97980, @SergeyKanzhelev) [SIG Node]

• Fix to recover CSI volumes from certain dangling attachments (#96617, @yuga711) [SIG Apps and Storage]

• Fix: azure file latency issue for metadata-heavy workloads (#97082, @andyzhangx) [SIG Cloud Provider and Storage]

• Fixed Cinder volume IDs on OpenStack Train (#96673, @jsafrane) [SIG Cloud Provider]

• Fixed FibreChannel volume plugin corrupting filesystems on detach of multipath volumes. (#97013, @jsafrane) [SIG Storage]

• Fixed a bug in kubelet that will saturate CPU utilization after containerd got restarted. (#97174, @hanlins) [SIG Node]

• Fixed bug in CPUManager with race on container map access (#97427, @klueska) [SIG Node]

• Fixed cleanup of block devices when /var/lib/kubelet is a symlink. (#96889, @jsafrane) [SIG Storage]

• GCE Internal LoadBalancer sync loop will now release the ILB IP address upon sync failure. An error in ILB forwarding rule creation will no longer leak IP addresses. (#97740, @prameshj) [SIG Cloud Provider and Network]

• Ignore update pod with no new images in alwaysPullImages admission controller (#96668, @pacoxu) [SIG Apps, Auth and Node]

• Kubeadm now installs version 3.4.13 of etcd when creating a cluster with v1.19 (#97244, @pacoxu) [SIG Cluster Lifecycle]

• Kubeadm: avoid detection of the container runtime for commands that do not need it (#97625, @pacoxu) [SIG Cluster Lifecycle]

• Kubeadm: fix a bug in the host memory detection code on 32bit Linux platforms (#97403, @abelbarrera15) [SIG Cluster Lifecycle]

• Kubeadm: fix a bug where "kubeadm upgrade" commands can fail if CoreDNS v1.8.0 is installed. (#97919, @neolit123) [SIG Cluster Lifecycle]

• Performance regression #97685 has been fixed. (#97860, @MikeSpreitzer) [SIG API Machinery]

• Remove deprecated --cleanup-ipvs flag of kube-proxy, and make --cleanup flag always to flush IPVS (#97336, @maaoBit) [SIG Network]

• The current version of the container image publicly exposed IP serving a /metrics endpoint to the Internet. The new version of the container image serves /metrics endpoint on a different port. (#97621, @vbannai) [SIG Cloud Provider]

• Use force unmount for NFS volumes if regular mount fails after 1 minute timeout (#96844, @gnufied) [SIG Storage]

• Users will see increase in time for deletion of pods and also guarantee that removal of pod from api server would mean deletion of all the resources from container runtime. (#92817, @kmala) [SIG Node]

• Using exec auth plugins with kubectl no longer results in warnings about constructing many client instances from the same exec auth config. (#97857, @liggitt) [SIG API Machinery and Auth]

• Warning about using a deprecated volume plugin is logged only once. (#96751, @jsafrane) [SIG Storage]

Other (Cleanup or Flake)

• Bump github.com/Azure/go-autorest/autorest to v0.11.12 (#97033, @patrickshan) [SIG API Machinery, CLI, Cloud Provider and Cluster Lifecycle]
• Delete deprecated mixed protocol annotation (#97096, @nilo19) [SIG Cloud Provider]
• Kube-proxy: Traffic from the cluster directed to ExternalIPs is always sent directly to the Service. (#96296, @aojea) [SIG Network and Testing]
• Kubeadm: fix a whitespace issue in the output of the "kubeadm join" command shown as the output of "kubeadm init" and "kubeadm token create --print-join-command" (#97413, @SataQiu) [SIG Cluster Lifecycle]
• Kubeadm: improve the error messaging when the user provides an invalid discovery token CA certificate hash. (#97290, @neolit123) [SIG Cluster Lifecycle]
• Migrate log messages in pkg/scheduler/{scheduler.go,factory.go} to structured logging (#97509, @aldudko) [SIG Scheduling]
• Migrate proxy/iptables/proxier.go logs to structured logging (#97678, @JornShen) [SIG Network]
• Migrate some scheduler log messages to structured logging (#97349, @aldudko) [SIG Scheduling]
• NONE (#97167, @geegeea) [SIG Node]
• NetworkPolicy validation framework optimizations for rapidly verifying CNI's work correctly across several pods and namespaces (#91592, @jayunit100) [SIG Network, Storage and Testing]
• Official support to build kubernetes with docker-machine / remote docker is removed. This change does not affect building kubernetes with docker locally. (#97618, @jherrera123) [SIG Release and Testing]
• Scheduler plugin validation now provides all errors detected instead of the first one. (#96745, @lingsamuel) [SIG Node, Scheduling and Testing]
• Storage related e2e testsuite redesign & cleanup (#96573, @Jiawei0227) [SIG Storage and Testing]
• The OIDC authenticator no longer waits 10 seconds before attempting to fetch the metadata required to verify tokens. (#97693, @enj) [SIG API Machinery and Auth]
• The AttachVolumeLimit feature gate that is GA since v1.17 is now removed. (#96539, @ialidzhikov) [SIG Storage]
• The CSINodeInfo feature gate that is GA since v1.17 is unconditionally enabled, and can no longer be specified via the --feature-gates argument. (#96561, @ialidzhikov) [SIG Apps, Auth, Scheduling, Storage and Testing]
• The deprecated feature gates RotateKubeletClientCertificate, AttachVolumeLimit, VolumePVCDataSource and EvenPodsSpread are now unconditionally enabled and can no longer be specified in component invocations. (#97306, @gavinfish) [SIG Node, Scheduling and Storage]
• ServiceNodeExclusion, NodeDisruptionExclusion and LegacyNodeRoleBehavior(locked to false) features have been promoted to GA. To prevent control plane nodes being added to load balancers automatically, upgrade users need to add "node.kubernetes.io/exclude-from-external-load-balancers" label to control plane nodes. (#97543, @pacoxu) [SIG API Machinery, Apps, Cloud Provider and Network]

Uncategorized

• Adding Brazilian Portuguese translation for kubectl (#61595, @cpanato) [SIG CLI]

Dependencies

Added

Nothing has changed.

Changed

• github.com/Azure/go-autorest/autorest: v0.11.1 → v0.11.12
• github.com/coredns/corefile-migration: v1.0.10 → v1.0.11
• github.com/golang/mock: v1.4.1 → v1.4.4
• github.com/google/cadvisor: v0.38.5 → v0.38.6
• github.com/heketi/heketi: c2e2a4a → v10.2.0+incompatible
• github.com/miekg/dns: v1.1.4 → v1.1.35
• k8s.io/system-validators: v1.2.0 → v1.3.0

Removed

• rsc.io/quote/v3: v3.1.0
• rsc.io/sampler: v1.3.0

1.2 - Kubernetes version and version skew support policy

This document describes the maximum version skew supported between various Kubernetes components. Specific cluster deployment tools may place additional restrictions on version skew.

Supported versions

Kubernetes versions are expressed as x.y.z, where x is the major version, y is the minor version, and z is the patch version, following Semantic Versioning terminology. For more information, see Kubernetes Release Versioning.

The Kubernetes project maintains release branches for the most recent three minor releases (1.22, 1.21, 1.20). Kubernetes 1.19 and newer receive approximately 1 year of patch support. Kubernetes 1.18 and older received approximately 9 months of patch support.

Applicable fixes, including security fixes, may be backported to those three release branches, depending on severity and feasibility. Patch releases are cut from those branches at a regular cadence, plus additional urgent releases, when required.

The Release Managers group owns this decision.

For more information, see the Kubernetes patch releases page.

Supported version skew

kube-apiserver

In highly-available (HA) clusters, the newest and oldest kube-apiserver instances must be within one minor version.

Example:

• newest kube-apiserver is at 1.22
• other kube-apiserver instances are supported at 1.22 and 1.21

kubelet

kubelet must not be newer than kube-apiserver, and may be up to two minor versions older.

Example:

• kube-apiserver is at 1.22
• kubelet is supported at 1.22, 1.21, and 1.20

Note: If version skew exists between kube-apiserver instances in an HA cluster, this narrows the allowed kubelet versions.

Example:

• kube-apiserver instances are at 1.22 and 1.21
• kubelet is supported at 1.21, and 1.20 (1.22 is not supported because that would be newer than the kube-apiserver instance at version 1.21)

kube-controller-manager, kube-scheduler, and cloud-controller-manager

kube-controller-manager, kube-scheduler, and cloud-controller-manager must not be newer than the kube-apiserver instances they communicate with. They are expected to match the kube-apiserver minor version, but may be up to one minor version older (to allow live upgrades).

Example:

• kube-apiserver is at 1.22
• kube-controller-manager, kube-scheduler, and cloud-controller-manager are supported at 1.22 and 1.21

Note: If version skew exists between kube-apiserver instances in an HA cluster, and these components can communicate with any kube-apiserver instance in the cluster (for example, via a load balancer), this narrows the allowed versions of these components.

Example:

• kube-apiserver instances are at 1.22 and 1.21
• kube-controller-manager, kube-scheduler, and cloud-controller-manager communicate with a load balancer that can route to any kube-apiserver instance
• kube-controller-manager, kube-scheduler, and cloud-controller-manager are supported at 1.21 (1.22 is not supported because that would be newer than the kube-apiserver instance at version 1.21)

kubectl

kubectl is supported within one minor version (older or newer) of kube-apiserver.

Example:

• kube-apiserver is at 1.22
• kubectl is supported at 1.23, 1.22, and 1.21

Note: If version skew exists between kube-apiserver instances in an HA cluster, this narrows the supported kubectl versions.

Example:

• kube-apiserver instances are at 1.22 and 1.21
• kubectl is supported at 1.22 and 1.21 (other versions would be more than one minor version skewed from one of the kube-apiserver components)

Supported component upgrade order

The supported version skew between components has implications on the order in which components must be upgraded. This section describes the order in which components must be upgraded to transition an existing cluster from version 1.21 to version 1.22.

kube-apiserver

Pre-requisites:

• In a single-instance cluster, the existing kube-apiserver instance is 1.21
• In an HA cluster, all kube-apiserver instances are at 1.21 or 1.22 (this ensures maximum skew of 1 minor version between the oldest and newest kube-apiserver instance)
• The kube-controller-manager, kube-scheduler, and cloud-controller-manager instances that communicate with this server are at version 1.21 (this ensures they are not newer than the existing API server version, and are within 1 minor version of the new API server version)
• kubelet instances on all nodes are at version 1.21 or 1.20 (this ensures they are not newer than the existing API server version, and are within 2 minor versions of the new API server version)
• Registered admission webhooks are able to handle the data the new kube-apiserver instance will send them:
  • ValidatingWebhookConfiguration and MutatingWebhookConfiguration objects are updated to include any new versions of REST resources added in 1.22 (or use the matchPolicy: Equivalent option available in v1.15+)
  • The webhooks are able to handle any new versions of REST resources that will be sent to them, and any new fields added to existing versions in 1.22

Upgrade kube-apiserver to 1.22

Note: Project policies for API deprecation and API change guidelines require kube-apiserver to not skip minor versions when upgrading, even in single-instance clusters.

kube-controller-manager, kube-scheduler, and cloud-controller-manager

Pre-requisites:

• The kube-apiserver instances these components communicate with are at 1.22 (in HA clusters in which these control plane components can communicate with any kube-apiserver instance in the cluster, all kube-apiserver instances must be upgraded before upgrading these components)

Upgrade kube-controller-manager, kube-scheduler, and cloud-controller-manager to 1.22

kubelet

Pre-requisites:

• The kube-apiserver instances the kubelet communicates with are at 1.22

Optionally upgrade kubelet instances to 1.22 (or they can be left at 1.21 or 1.20)

Note: Before performing a minor version kubelet upgrade, drain pods from that node. In-place minor version kubelet upgrades are not supported.

Warning:

Running a cluster with kubelet instances that are persistently two minor versions behind kube-apiserver is not recommended:

• they must be upgraded within one minor version of kube-apiserver before the control plane can be upgraded
• it increases the likelihood of running kubelet versions older than the three maintained minor releases

kube-proxy

• kube-proxy must be the same minor version as kubelet on the node.
• kube-proxy must not be newer than kube-apiserver.
• kube-proxy must be at most two minor versions older than kube-apiserver.

Example:

If kube-proxy version is 1.20:

• kubelet version must be at the same minor version as 1.20.
• kube-apiserver version must be between 1.20 and 1.22, inclusive.

2 - Learning environment

kind

kind lets you run Kubernetes on your local computer. This tool requires that you have Docker installed and configured.

The kind Quick Start page shows you what you need to do to get up and running with kind.

minikube

Like kind, minikube is a tool that lets you run Kubernetes locally. minikube runs a single-node Kubernetes cluster on your personal computer (including Windows, macOS and Linux PCs) so that you can try out Kubernetes, or for daily development work.

You can follow the official Get Started! guide if your focus is on getting the tool installed.

3 - Production environment

A production-quality Kubernetes cluster requires planning and preparation. If your Kubernetes cluster is to run critical workloads, it must be configured to be resilient. This page explains steps you can take to set up a production-ready cluster, or to uprate an existing cluster for production use. If you're already familiar with production setup and want the links, skip to What's next.

Production considerations

Typically, a production Kubernetes cluster environment has more requirements than a personal learning, development, or test environment Kubernetes. A production environment may require secure access by many users, consistent availability, and the resources to adapt to changing demands.

As you decide where you want your production Kubernetes environment to live (on premises or in a cloud) and the amount of management you want to take on or hand to others, consider how your requirements for a Kubernetes cluster are influenced by the following issues:

• Availability: A single-machine Kubernetes learning environment has a single point of failure. Creating a highly available cluster means considering:

  • Separating the control plane from the worker nodes.
  • Replicating the control plane components on multiple nodes.
  • Load balancing traffic to the cluster’s API server.
  • Having enough worker nodes available, or able to quickly become available, as changing workloads warrant it.

• Scale: If you expect your production Kubernetes environment to receive a stable amount of demand, you might be able to set up for the capacity you need and be done. However, if you expect demand to grow over time or change dramatically based on things like season or special events, you need to plan how to scale to relieve increased pressure from more requests to the control plane and worker nodes or scale down to reduce unused resources.

• Security and access management: You have full admin privileges on your own Kubernetes learning cluster. But shared clusters with important workloads, and more than one or two users, require a more refined approach to who and what can access cluster resources. You can use role-based access control (RBAC) and other security mechanisms to make sure that users and workloads can get access to the resources they need, while keeping workloads, and the cluster itself, secure. You can set limits on the resources that users and workloads can access by managing policies and container resources.

Before building a Kubernetes production environment on your own, consider handing off some or all of this job to Turnkey Cloud Solutions providers or other Kubernetes Partners. Options include:

• Serverless: Just run workloads on third-party equipment without managing a cluster at all. You will be charged for things like CPU usage, memory, and disk requests.
• Managed control plane: Let the provider manage the scale and availability of the cluster's control plane, as well as handle patches and upgrades.
• Managed worker nodes: Configure pools of nodes to meet your needs, then the provider makes sure those nodes are available and ready to implement upgrades when needed.
• Integration: There are providers that integrate Kubernetes with other services you may need, such as storage, container registries, authentication methods, and development tools.

Whether you build a production Kubernetes cluster yourself or work with partners, review the following sections to evaluate your needs as they relate to your cluster’s control plane, worker nodes, user access, and workload resources.

Production cluster setup

In a production-quality Kubernetes cluster, the control plane manages the cluster from services that can be spread across multiple computers in different ways. Each worker node, however, represents a single entity that is configured to run Kubernetes pods.

Production control plane

The simplest Kubernetes cluster has the entire control plane and worker node services running on the same machine. You can grow that environment by adding worker nodes, as reflected in the diagram illustrated in Kubernetes Components. If the cluster is meant to be available for a short period of time, or can be discarded if something goes seriously wrong, this might meet your needs.

If you need a more permanent, highly available cluster, however, you should consider ways of extending the control plane. By design, one-machine control plane services running on a single machine are not highly available. If keeping the cluster up and running and ensuring that it can be repaired if something goes wrong is important, consider these steps:

• Choose deployment tools: You can deploy a control plane using tools such as kubeadm, kops, and kubespray. See Installing Kubernetes with deployment tools to learn tips for production-quality deployments using each of those deployment methods. Different Container Runtimes are available to use with your deployments.
• Manage certificates: Secure communications between control plane services are implemented using certificates. Certificates are automatically generated during deployment or you can generate them using your own certificate authority. See PKI certificates and requirements for details.
• Configure load balancer for apiserver: Configure a load balancer to distribute external API requests to the apiserver service instances running on different nodes. See Create an External Load Balancer for details.
• Separate and backup etcd service: The etcd services can either run on the same machines as other control plane services or run on separate machines, for extra security and availability. Because etcd stores cluster configuration data, backing up the etcd database should be done regularly to ensure that you can repair that database if needed. See the etcd FAQ for details on configuring and using etcd. See Operating etcd clusters for Kubernetes and Set up a High Availability etcd cluster with kubeadm for details.
• Create multiple control plane systems: For high availability, the control plane should not be limited to a single machine. If the control plane services are run by an init service (such as systemd), each service should run on at least three machines. However, running control plane services as pods in Kubernetes ensures that the replicated number of services that you request will always be available. The scheduler should be fault tolerant, but not highly available. Some deployment tools set up Raft consensus algorithm to do leader election of Kubernetes services. If the primary goes away, another service elects itself and take over.
• Span multiple zones: If keeping your cluster available at all times is critical, consider creating a cluster that runs across multiple data centers, referred to as zones in cloud environments. Groups of zones are referred to as regions. By spreading a cluster across multiple zones in the same region, it can improve the chances that your cluster will continue to function even if one zone becomes unavailable. See Running in multiple zones for details.
• Manage on-going features: If you plan to keep your cluster over time, there are tasks you need to do to maintain its health and security. For example, if you installed with kubeadm, there are instructions to help you with Certificate Management and Upgrading kubeadm clusters. See Administer a Cluster for a longer list of Kubernetes administrative tasks.

To learn about available options when you run control plane services, see kube-apiserver, kube-controller-manager, and kube-scheduler component pages. For highly available control plane examples, see Options for Highly Available topology, Creating Highly Available clusters with kubeadm, and Operating etcd clusters for Kubernetes. See Backing up an etcd cluster for information on making an etcd backup plan.

Production worker nodes

Production-quality workloads need to be resilient and anything they rely on needs to be resilient (such as CoreDNS). Whether you manage your own control plane or have a cloud provider do it for you, you still need to consider how you want to manage your worker nodes (also referred to simply as nodes).

• Configure nodes: Nodes can be physical or virtual machines. If you want to create and manage your own nodes, you can install a supported operating system, then add and run the appropriate Node services. Consider:
  • The demands of your workloads when you set up nodes by having appropriate memory, CPU, and disk speed and storage capacity available.
  • Whether generic computer systems will do or you have workloads that need GPU processors, Windows nodes, or VM isolation.
• Validate nodes: See Valid node setup for information on how to ensure that a node meets the requirements to join a Kubernetes cluster.
• Add nodes to the cluster: If you are managing your own cluster you can add nodes by setting up your own machines and either adding them manually or having them register themselves to the cluster’s apiserver. See the Nodes section for information on how to set up Kubernetes to add nodes in these ways.
• Add Windows nodes to the cluster: Kubernetes offers support for Windows worker nodes, allowing you to run workloads implemented in Windows containers. See Windows in Kubernetes for details.
• Scale nodes: Have a plan for expanding the capacity your cluster will eventually need. See Considerations for large clusters to help determine how many nodes you need, based on the number of pods and containers you need to run. If you are managing nodes yourself, this can mean purchasing and installing your own physical equipment.
• Autoscale nodes: Most cloud providers support Cluster Autoscaler to replace unhealthy nodes or grow and shrink the number of nodes as demand requires. See the Frequently Asked Questions for how the autoscaler works and Deployment for how it is implemented by different cloud providers. For on-premises, there are some virtualization platforms that can be scripted to spin up new nodes based on demand.
• Set up node health checks: For important workloads, you want to make sure that the nodes and pods running on those nodes are healthy. Using the Node Problem Detector daemon, you can ensure your nodes are healthy.

Production user management

In production, you may be moving from a model where you or a small group of people are accessing the cluster to where there may potentially be dozens or hundreds of people. In a learning environment or platform prototype, you might have a single administrative account for everything you do. In production, you will want more accounts with different levels of access to different namespaces.

Taking on a production-quality cluster means deciding how you want to selectively allow access by other users. In particular, you need to select strategies for validating the identities of those who try to access your cluster (authentication) and deciding if they have permissions to do what they are asking (authorization):

• Authentication: The apiserver can authenticate users using client certificates, bearer tokens, an authenticating proxy, or HTTP basic auth. You can choose which authentication methods you want to use. Using plugins, the apiserver can leverage your organization’s existing authentication methods, such as LDAP or Kerberos. See Authentication for a description of these different methods of authenticating Kubernetes users.
• Authorization: When you set out to authorize your regular users, you will probably choose between RBAC and ABAC authorization. See Authorization Overview to review different modes for authorizing user accounts (as well as service account access to your cluster):
  • Role-based access control (RBAC): Lets you assign access to your cluster by allowing specific sets of permissions to authenticated users. Permissions can be assigned for a specific namespace (Role) or across the entire cluster (ClusterRole). Then using RoleBindings and ClusterRoleBindings, those permissions can be attached to particular users.
  • Attribute-based access control (ABAC): Lets you create policies based on resource attributes in the cluster and will allow or deny access based on those attributes. Each line of a policy file identifies versioning properties (apiVersion and kind) and a map of spec properties to match the subject (user or group), resource property, non-resource property (/version or /apis), and readonly. See Examples for details.

As someone setting up authentication and authorization on your production Kubernetes cluster, here are some things to consider:

• Set the authorization mode: When the Kubernetes API server (kube-apiserver) starts, the supported authentication modes must be set using the --authorization-mode flag. For example, that flag in the kube-adminserver.yaml file (in /etc/kubernetes/manifests) could be set to Node,RBAC. This would allow Node and RBAC authorization for authenticated requests.
• Create user certificates and role bindings (RBAC): If you are using RBAC authorization, users can create a CertificateSigningRequest (CSR) that can be signed by the cluster CA. Then you can bind Roles and ClusterRoles to each user. See Certificate Signing Requests for details.
• Create policies that combine attributes (ABAC): If you are using ABAC authorization, you can assign combinations of attributes to form policies to authorize selected users or groups to access particular resources (such as a pod), namespace, or apiGroup. For more information, see Examples.
• Consider Admission Controllers: Additional forms of authorization for requests that can come in through the API server include Webhook Token Authentication. Webhooks and other special authorization types need to be enabled by adding Admission Controllers to the API server.

Set limits on workload resources

Demands from production workloads can cause pressure both inside and outside of the Kubernetes control plane. Consider these items when setting up for the needs of your cluster's workloads:

• Set namespace limits: Set per-namespace quotas on things like memory and CPU. See Manage Memory, CPU, and API Resources for details. You can also set Hierarchical Namespaces for inheriting limits.
• Prepare for DNS demand: If you expect workloads to massively scale up, your DNS service must be ready to scale up as well. See Autoscale the DNS service in a Cluster.
• Create additional service accounts: User accounts determine what users can do on a cluster, while a service account defines pod access within a particular namespace. By default, a pod takes on the default service account from its namespace. See Managing Service Accounts for information on creating a new service account. For example, you might want to:
  • Add secrets that a pod could use to pull images from a particular container registry. See Configure Service Accounts for Pods for an example.
  • Assign RBAC permissions to a service account. See ServiceAccount permissions for details.

What's next

• Decide if you want to build your own production Kubernetes or obtain one from available Turnkey Cloud Solutions or Kubernetes Partners.
• If you choose to build your own cluster, plan how you want to handle certificates and set up high availability for features such as etcd and the API server.
• Choose from kubeadm, kops or Kubespray deployment methods.
• Configure user management by determining your Authentication and Authorization methods.
• Prepare for application workloads by setting up resource limits, DNS autoscaling and service accounts.

3.1 - Container runtimes

You need to install a container runtime into each node in the cluster so that Pods can run there. This page outlines what is involved and describes related tasks for setting up nodes.

This page lists details for using several common container runtimes with Kubernetes, on Linux:

• containerd
• CRI-O
• Docker

Note: For other operating systems, look for documentation specific to your platform.

Cgroup drivers

Control groups are used to constrain resources that are allocated to processes.

When systemd is chosen as the init system for a Linux distribution, the init process generates and consumes a root control group (cgroup) and acts as a cgroup manager. Systemd has a tight integration with cgroups and allocates a cgroup per systemd unit. It's possible to configure your container runtime and the kubelet to use cgroupfs. Using cgroupfs alongside systemd means that there will be two different cgroup managers.

A single cgroup manager simplifies the view of what resources are being allocated and will by default have a more consistent view of the available and in-use resources. When there are two cgroup managers on a system, you end up with two views of those resources. In the field, people have reported cases where nodes that are configured to use cgroupfs for the kubelet and Docker, but systemd for the rest of the processes, become unstable under resource pressure.

Changing the settings such that your container runtime and kubelet use systemd as the cgroup driver stabilized the system. To configure this for Docker, set native.cgroupdriver=systemd.

Caution:

Changing the cgroup driver of a Node that has joined a cluster is a sensitive operation. If the kubelet has created Pods using the semantics of one cgroup driver, changing the container runtime to another cgroup driver can cause errors when trying to re-create the Pod sandbox for such existing Pods. Restarting the kubelet may not solve such errors.

If you have automation that makes it feasible, replace the node with another using the updated configuration, or reinstall it using automation.

Migrating to the systemd driver in kubeadm managed clusters

Follow this Migration guide if you wish to migrate to the systemd cgroup driver in existing kubeadm managed clusters.

Container runtimes

Caution: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects. This page follows CNCF website guidelines by listing projects alphabetically. To add a project to this list, read the content guide before submitting a change.

containerd

This section contains the necessary steps to use containerd as CRI runtime.

Use the following commands to install Containerd on your system:

Install and configure prerequisites:

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# Setup required sysctl params, these persist across reboots.
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

# Apply sysctl params without reboot
sudo sysctl --system

Install containerd:

• Linux
• Windows (PowerShell)

Using the systemd cgroup driver

To use the systemd cgroup driver in /etc/containerd/config.toml with runc, set

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

If you apply this change make sure to restart containerd again:

sudo systemctl restart containerd

When using kubeadm, manually configure the cgroup driver for kubelet.

CRI-O

This section contains the necessary steps to install CRI-O as a container runtime.

Use the following commands to install CRI-O on your system:

Note: The CRI-O major and minor versions must match the Kubernetes major and minor versions. For more information, see the CRI-O compatibility matrix.

Install and configure prerequisites:

# Create the .conf file to load the modules at bootup
cat <<EOF | sudo tee /etc/modules-load.d/crio.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# Set up required sysctl params, these persist across reboots.
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sudo sysctl --system

• Debian
• Ubuntu
• CentOS
• openSUSE Tumbleweed
• Fedora

cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /
EOF
cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list
deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /
EOF

curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers.gpg add -
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers.gpg add -

sudo apt-get update
sudo apt-get install cri-o cri-o-runc

cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /
EOF
cat <<EOF | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list
deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /
EOF

curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers.gpg add -
curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/Release.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers-cri-o.gpg add -

sudo apt-get update
sudo apt-get install cri-o cri-o-runc

sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/devel:kubic:libcontainers:stable.repo
sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo
sudo yum install cri-o

sudo zypper install cri-o

sudo dnf module list cri-o

sudo dnf module enable cri-o:$VERSION
sudo dnf install cri-o

Start CRI-O:

sudo systemctl daemon-reload
sudo systemctl enable crio --now

Refer to the CRI-O installation guide for more information.

cgroup driver

CRI-O uses the systemd cgroup driver per default. To switch to the cgroupfs cgroup driver, either edit /etc/crio/crio.conf or place a drop-in configuration in /etc/crio/crio.conf.d/02-cgroup-manager.conf, for example:

[crio.runtime]
conmon_cgroup = "pod"
cgroup_manager = "cgroupfs"

Please also note the changed conmon_cgroup, which has to be set to the value pod when using CRI-O with cgroupfs. It is generally necessary to keep the cgroup driver configuration of the kubelet (usually done via kubeadm) and CRI-O in sync.

Docker

1. On each of your nodes, install the Docker for your Linux distribution as per Install Docker Engine. You can find the latest validated version of Docker in this dependencies file.

2. Configure the Docker daemon, in particular to use systemd for the management of the container’s cgroups.

   sudo mkdir /etc/docker
   cat <<EOF | sudo tee /etc/docker/daemon.json
   {
     "exec-opts": ["native.cgroupdriver=systemd"],
     "log-driver": "json-file",
     "log-opts": {
       "max-size": "100m"
     },
     "storage-driver": "overlay2"
   }
   EOF
   

   Note: overlay2 is the preferred storage driver for systems running Linux kernel version 4.0 or higher, or RHEL or CentOS using version 3.10.0-514 and above.

3. Restart Docker and enable on boot:

   sudo systemctl enable docker
   sudo systemctl daemon-reload
   sudo systemctl restart docker
   

Note:

For more information refer to

• Configure the Docker daemon
• Control Docker with systemd

3.2 - Installing Kubernetes with deployment tools

3.2.1 - Bootstrapping clusters with kubeadm

3.2.1.1 - Installing kubeadm

This page shows how to install the kubeadm toolbox. For information how to create a cluster with kubeadm once you have performed this installation process, see the Using kubeadm to Create a Cluster page.

Before you begin

• A compatible Linux host. The Kubernetes project provides generic instructions for Linux distributions based on Debian and Red Hat, and those distributions without a package manager.
• 2 GB or more of RAM per machine (any less will leave little room for your apps).
• 2 CPUs or more.
• Full network connectivity between all machines in the cluster (public or private network is fine).
• Unique hostname, MAC address, and product_uuid for every node. See here for more details.
• Certain ports are open on your machines. See here for more details.
• Swap disabled. You MUST disable swap in order for the kubelet to work properly.

Verify the MAC address and product_uuid are unique for every node

• You can get the MAC address of the network interfaces using the command ip link or ifconfig -a
• The product_uuid can be checked by using the command sudo cat /sys/class/dmi/id/product_uuid

It is very likely that hardware devices will have unique addresses, although some virtual machines may have identical values. Kubernetes uses these values to uniquely identify the nodes in the cluster. If these values are not unique to each node, the installation process may fail.

Check network adapters

If you have more than one network adapter, and your Kubernetes components are not reachable on the default route, we recommend you add IP route(s) so Kubernetes cluster addresses go via the appropriate adapter.

Letting iptables see bridged traffic

Make sure that the br_netfilter module is loaded. This can be done by running lsmod | grep br_netfilter. To load it explicitly call sudo modprobe br_netfilter.

As a requirement for your Linux Node's iptables to correctly see bridged traffic, you should ensure net.bridge.bridge-nf-call-iptables is set to 1 in your sysctl config, e.g.

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

For more details please see the Network Plugin Requirements page.

Check required ports

Control-plane node(s)

Worker node(s)

† Default port range for NodePort Services.

Any port numbers marked with * are overridable, so you will need to ensure any custom ports you provide are also open.

Although etcd ports are included in control-plane nodes, you can also host your own etcd cluster externally or on custom ports.

The pod network plugin you use (see below) may also require certain ports to be open. Since this differs with each pod network plugin, please see the documentation for the plugins about what port(s) those need.

Installing runtime

To run containers in Pods, Kubernetes uses a container runtime.

• Linux nodes
• other operating systems

Installing kubeadm, kubelet and kubectl

You will install these packages on all of your machines:

• kubeadm: the command to bootstrap the cluster.

• kubelet: the component that runs on all of the machines in your cluster and does things like starting pods and containers.

• kubectl: the command line util to talk to your cluster.

kubeadm will not install or manage kubelet or kubectl for you, so you will need to ensure they match the version of the Kubernetes control plane you want kubeadm to install for you. If you do not, there is a risk of a version skew occurring that can lead to unexpected, buggy behaviour. However, one minor version skew between the kubelet and the control plane is supported, but the kubelet version may never exceed the API server version. For example, the kubelet running 1.7.0 should be fully compatible with a 1.8.0 API server, but not vice versa.

For information about installing kubectl, see Install and set up kubectl.

Warning: These instructions exclude all Kubernetes packages from any system upgrades. This is because kubeadm and Kubernetes require special attention to upgrade.

For more information on version skews, see:

• Kubernetes version and version-skew policy
• Kubeadm-specific version skew policy

• Debian-based distributions
• Red Hat-based distributions
• Without a package manager

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

# Set SELinux in permissive mode (effectively disabling it)
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

sudo systemctl enable --now kubelet

CNI_VERSION="v0.8.2"
sudo mkdir -p /opt/cni/bin
curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz" | sudo tar -C /opt/cni/bin -xz

DOWNLOAD_DIR=/usr/local/bin
sudo mkdir -p $DOWNLOAD_DIR

CRICTL_VERSION="v1.17.0"
curl -L "https://github.com/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-amd64.tar.gz" | sudo tar -C $DOWNLOAD_DIR -xz

RELEASE="$(curl -sSL https://dl.k8s.io/release/stable.txt)"
cd $DOWNLOAD_DIR
sudo curl -L --remote-name-all https://storage.googleapis.com/kubernetes-release/release/${RELEASE}/bin/linux/amd64/{kubeadm,kubelet,kubectl}
sudo chmod +x {kubeadm,kubelet,kubectl}

RELEASE_VERSION="v0.4.0"
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /etc/systemd/system/kubelet.service
sudo mkdir -p /etc/systemd/system/kubelet.service.d
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/kubepkg/templates/latest/deb/kubeadm/10-kubeadm.conf" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

systemctl enable --now kubelet

The kubelet is now restarting every few seconds, as it waits in a crashloop for kubeadm to tell it what to do.

Configuring a cgroup driver

Both the container runtime and the kubelet have a property called "cgroup driver", which is important for the management of cgroups on Linux machines.

Warning:

Matching the container runtime and kubelet cgroup drivers is required or otherwise the kubelet process will fail.

See Configuring a cgroup driver for more details.

Troubleshooting

If you are running into difficulties with kubeadm, please consult our troubleshooting docs.

What's next

• Using kubeadm to Create a Cluster

3.2.1.2 - Troubleshooting kubeadm

As with any program, you might run into an error installing or running kubeadm. This page lists some common failure scenarios and have provided steps that can help you understand and fix the problem.

If your problem is not listed below, please follow the following steps:

• If you think your problem is a bug with kubeadm:

  • Go to github.com/kubernetes/kubeadm and search for existing issues.
  • If no issue exists, please open one and follow the issue template.

• If you are unsure about how kubeadm works, you can ask on Slack in #kubeadm, or open a question on StackOverflow. Please include relevant tags like #kubernetes and #kubeadm so folks can help you.

Not possible to join a v1.18 Node to a v1.17 cluster due to missing RBAC

In v1.18 kubeadm added prevention for joining a Node in the cluster if a Node with the same name already exists. This required adding RBAC for the bootstrap-token user to be able to GET a Node object.

However this causes an issue where kubeadm join from v1.18 cannot join a cluster created by kubeadm v1.17.

To workaround the issue you have two options:

Execute kubeadm init phase bootstrap-token on a control-plane node using kubeadm v1.18. Note that this enables the rest of the bootstrap-token permissions as well.

or

Apply the following RBAC manually using kubectl apply -f ...:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubeadm:get-nodes
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubeadm:get-nodes
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubeadm:get-nodes
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:kubeadm:default-node-token

ebtables or some similar executable not found during installation

If you see the following warnings while running kubeadm init

[preflight] WARNING: ebtables not found in system path
[preflight] WARNING: ethtool not found in system path

Then you may be missing ebtables, ethtool or a similar executable on your node. You can install them with the following commands:

• For Ubuntu/Debian users, run apt install ebtables ethtool.
• For CentOS/Fedora users, run yum install ebtables ethtool.

kubeadm blocks waiting for control plane during installation

If you notice that kubeadm init hangs after printing out the following line:

[apiclient] Created API client, waiting for the control plane to become ready

This may be caused by a number of problems. The most common are:

• network connection problems. Check that your machine has full network connectivity before continuing.

• the default cgroup driver configuration for the kubelet differs from that used by Docker. Check the system log file (e.g. /var/log/message) or examine the output from journalctl -u kubelet. If you see something like the following:

  error: failed to run Kubelet: failed to create kubelet:
  misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"
  

  There are two common ways to fix the cgroup driver problem:

  1. Install Docker again following instructions here.

  2. Change the kubelet config to match the Docker cgroup driver manually, you can refer to Configure cgroup driver used by kubelet on control-plane node

• control plane Docker containers are crashlooping or hanging. You can check this by running docker ps and investigating each container by running docker logs.

kubeadm blocks when removing managed containers

The following could happen if Docker halts and does not remove any Kubernetes-managed containers:

sudo kubeadm reset

[preflight] Running pre-flight checks
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Removing kubernetes-managed containers
(block)

A possible solution is to restart the Docker service and then re-run kubeadm reset:

sudo systemctl restart docker.service
sudo kubeadm reset

Inspecting the logs for docker may also be useful:

journalctl -u docker

Pods in RunContainerError, CrashLoopBackOff or Error state

Right after kubeadm init there should not be any pods in these states.

• If there are pods in one of these states right after kubeadm init, please open an issue in the kubeadm repo. coredns (or kube-dns) should be in the Pending state until you have deployed the network add-on.
• If you see Pods in the RunContainerError, CrashLoopBackOff or Error state after deploying the network add-on and nothing happens to coredns (or kube-dns), it's very likely that the Pod Network add-on that you installed is somehow broken. You might have to grant it more RBAC privileges or use a newer version. Please file an issue in the Pod Network providers' issue tracker and get the issue triaged there.
• If you install a version of Docker older than 1.12.1, remove the MountFlags=slave option when booting dockerd with systemd and restart docker. You can see the MountFlags in /usr/lib/systemd/system/docker.service. MountFlags can interfere with volumes mounted by Kubernetes, and put the Pods in CrashLoopBackOff state. The error happens when Kubernetes does not find var/run/secrets/kubernetes.io/serviceaccount files.

coredns is stuck in the Pending state

This is expected and part of the design. kubeadm is network provider-agnostic, so the admin should install the pod network add-on of choice. You have to install a Pod Network before CoreDNS may be deployed fully. Hence the Pending state before the network is set up.

HostPort services do not work

The HostPort and HostIP functionality is available depending on your Pod Network provider. Please contact the author of the Pod Network add-on to find out whether HostPort and HostIP functionality are available.

Calico, Canal, and Flannel CNI providers are verified to support HostPort.

For more information, see the CNI portmap documentation.

If your network provider does not support the portmap CNI plugin, you may need to use the NodePort feature of services or use HostNetwork=true.

Pods are not accessible via their Service IP

• Many network add-ons do not yet enable hairpin mode which allows pods to access themselves via their Service IP. This is an issue related to CNI. Please contact the network add-on provider to get the latest status of their support for hairpin mode.

• If you are using VirtualBox (directly or via Vagrant), you will need to ensure that hostname -i returns a routable IP address. By default the first interface is connected to a non-routable host-only network. A work around is to modify /etc/hosts, see this Vagrantfile for an example.

TLS certificate errors

The following error indicates a possible certificate mismatch.

# kubectl get pods
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")

• Verify that the $HOME/.kube/config file contains a valid certificate, and regenerate a certificate if necessary. The certificates in a kubeconfig file are base64 encoded. The base64 --decode command can be used to decode the certificate and openssl x509 -text -noout can be used for viewing the certificate information.

• Unset the KUBECONFIG environment variable using:

  unset KUBECONFIG
  

  Or set it to the default KUBECONFIG location:

  export KUBECONFIG=/etc/kubernetes/admin.conf
  

• Another workaround is to overwrite the existing kubeconfig for the "admin" user:

  mv  $HOME/.kube $HOME/.kube.bak
  mkdir $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
  

Default NIC When using flannel as the pod network in Vagrant

The following error might indicate that something was wrong in the pod network:

Error from server (NotFound): the server could not find the requested resource

• If you're using flannel as the pod network inside Vagrant, then you will have to specify the default interface name for flannel.

  Vagrant typically assigns two interfaces to all VMs. The first, for which all hosts are assigned the IP address 10.0.2.15, is for external traffic that gets NATed.

  This may lead to problems with flannel, which defaults to the first interface on a host. This leads to all hosts thinking they have the same public IP address. To prevent this, pass the --iface eth1 flag to flannel so that the second interface is chosen.

Non-public IP used for containers

In some situations kubectl logs and kubectl run commands may return with the following errors in an otherwise functional cluster:

Error from server: Get https://10.19.0.41:10250/containerLogs/default/mysql-ddc65b868-glc5m/mysql: dial tcp 10.19.0.41:10250: getsockopt: no route to host

• This may be due to Kubernetes using an IP that can not communicate with other IPs on the seemingly same subnet, possibly by policy of the machine provider.

• DigitalOcean assigns a public IP to eth0 as well as a private one to be used internally as anchor for their floating IP feature, yet kubelet will pick the latter as the node's InternalIP instead of the public one.

  Use ip addr show to check for this scenario instead of ifconfig because ifconfig will not display the offending alias IP address. Alternatively an API endpoint specific to DigitalOcean allows to query for the anchor IP from the droplet:

  curl http://169.254.169.254/metadata/v1/interfaces/public/0/anchor_ipv4/address
  

  The workaround is to tell kubelet which IP to use using --node-ip. When using DigitalOcean, it can be the public one (assigned to eth0) or the private one (assigned to eth1) should you want to use the optional private network. The KubeletExtraArgs section of the kubeadm NodeRegistrationOptions structure can be used for this.

  Then restart kubelet:

  systemctl daemon-reload
  systemctl restart kubelet
  

coredns pods have CrashLoopBackOff or Error state

If you have nodes that are running SELinux with an older version of Docker you might experience a scenario where the coredns pods are not starting. To solve that you can try one of the following options:

• Upgrade to a newer version of Docker.

• Disable SELinux.

• Modify the coredns deployment to set allowPrivilegeEscalation to true:

kubectl -n kube-system get deployment coredns -o yaml | \
  sed 's/allowPrivilegeEscalation: false/allowPrivilegeEscalation: true/g' | \
  kubectl apply -f -

Another cause for CoreDNS to have CrashLoopBackOff is when a CoreDNS Pod deployed in Kubernetes detects a loop. A number of workarounds are available to avoid Kubernetes trying to restart the CoreDNS Pod every time CoreDNS detects the loop and exits.

Warning: Disabling SELinux or setting allowPrivilegeEscalation to true can compromise the security of your cluster.

etcd pods restart continually

If you encounter the following error:

rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:110: decoding init error from pipe caused \"read parent: connection reset by peer\""

this issue appears if you run CentOS 7 with Docker 1.13.1.84. This version of Docker can prevent the kubelet from executing into the etcd container.

To work around the issue, choose one of these options:

• Roll back to an earlier version of Docker, such as 1.13.1-75

yum downgrade docker-1.13.1-75.git8633870.el7.centos.x86_64 docker-client-1.13.1-75.git8633870.el7.centos.x86_64 docker-common-1.13.1-75.git8633870.el7.centos.x86_64

• Install one of the more recent recommended versions, such as 18.06:

sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install docker-ce-18.06.1.ce-3.el7.x86_64

Not possible to pass a comma separated list of values to arguments inside a --component-extra-args flag

kubeadm init flags such as --component-extra-args allow you to pass custom arguments to a control-plane component like the kube-apiserver. However, this mechanism is limited due to the underlying type used for parsing the values (mapStringString).

If you decide to pass an argument that supports multiple, comma-separated values such as --apiserver-extra-args "enable-admission-plugins=LimitRanger,NamespaceExists" this flag will fail with flag: malformed pair, expect string=string. This happens because the list of arguments for --apiserver-extra-args expects key=value pairs and in this case NamespacesExists is considered as a key that is missing a value.

Alternatively, you can try separating the key=value pairs like so: --apiserver-extra-args "enable-admission-plugins=LimitRanger,enable-admission-plugins=NamespaceExists" but this will result in the key enable-admission-plugins only having the value of NamespaceExists.

A known workaround is to use the kubeadm configuration file.

kube-proxy scheduled before node is initialized by cloud-controller-manager

In cloud provider scenarios, kube-proxy can end up being scheduled on new worker nodes before the cloud-controller-manager has initialized the node addresses. This causes kube-proxy to fail to pick up the node's IP address properly and has knock-on effects to the proxy function managing load balancers.

The following error can be seen in kube-proxy Pods:

server.go:610] Failed to retrieve node IP: host IP unknown; known addresses: []
proxier.go:340] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP

A known solution is to patch the kube-proxy DaemonSet to allow scheduling it on control-plane nodes regardless of their conditions, keeping it off of other nodes until their initial guarding conditions abate:

kubectl -n kube-system patch ds kube-proxy -p='{ "spec": { "template": { "spec": { "tolerations": [ { "key": "CriticalAddonsOnly", "operator": "Exists" }, { "effect": "NoSchedule", "key": "node-role.kubernetes.io/master" } ] } } } }'

The tracking issue for this problem is here.

The NodeRegistration.Taints field is omitted when marshalling kubeadm configuration

Note: This issue only applies to tools that marshal kubeadm types (e.g. to a YAML configuration file). It will be fixed in kubeadm API v1beta2.

By default, kubeadm applies the node-role.kubernetes.io/master:NoSchedule taint to control-plane nodes. If you prefer kubeadm to not taint the control-plane node, and set InitConfiguration.NodeRegistration.Taints to an empty slice, the field will be omitted when marshalling. When the field is omitted, kubeadm applies the default taint.

There are at least two workarounds:

1. Use the node-role.kubernetes.io/master:PreferNoSchedule taint instead of an empty slice. Pods will get scheduled on masters, unless other nodes have capacity.

2. Remove the taint after kubeadm init exits:

kubectl taint nodes NODE_NAME node-role.kubernetes.io/master:NoSchedule-

/usr is mounted read-only on nodes

On Linux distributions such as Fedora CoreOS or Flatcar Container Linux, the directory /usr is mounted as a read-only filesystem. For flex-volume support, Kubernetes components like the kubelet and kube-controller-manager use the default path of /usr/libexec/kubernetes/kubelet-plugins/volume/exec/, yet the flex-volume directory must be writeable for the feature to work.

To workaround this issue you can configure the flex-volume directory using the kubeadm configuration file.

On the primary control-plane Node (created using kubeadm init) pass the following file using --config:

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration:
  kubeletExtraArgs:
    volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
controllerManager:
  extraArgs:
    flex-volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"

On joining Nodes:

apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
nodeRegistration:
  kubeletExtraArgs:
    volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"

Alternatively, you can modify /etc/fstab to make the /usr mount writeable, but please be advised that this is modifying a design principle of the Linux distribution.

kubeadm upgrade plan prints out context deadline exceeded error message

This error message is shown when upgrading a Kubernetes cluster with kubeadm in the case of running an external etcd. This is not a critical bug and happens because older versions of kubeadm perform a version check on the external etcd cluster. You can proceed with kubeadm upgrade apply ....

This issue is fixed as of version 1.19.

kubeadm reset unmounts /var/lib/kubelet

If /var/lib/kubelet is being mounted, performing a kubeadm reset will effectively unmount it.

To workaround the issue, re-mount the /var/lib/kubelet directory after performing the kubeadm reset operation.

This is a regression introduced in kubeadm 1.15. The issue is fixed in 1.20.

Cannot use the metrics-server securely in a kubeadm cluster

In a kubeadm cluster, the metrics-server can be used insecurely by passing the --kubelet-insecure-tls to it. This is not recommended for production clusters.

If you want to use TLS between the metrics-server and the kubelet there is a problem, since kubeadm deploys a self-signed serving certificate for the kubelet. This can cause the following errors on the side of the metrics-server:

x509: certificate signed by unknown authority
x509: certificate is valid for IP-foo not IP-bar

See Enabling signed kubelet serving certificates to understand how to configure the kubelets in a kubeadm cluster to have properly signed serving certificates.

Also see How to run the metrics-server securely.

3.2.1.3 - Creating a cluster with kubeadm

Using kubeadm, you can create a minimum viable Kubernetes cluster that conforms to best practices. In fact, you can use kubeadm to set up a cluster that will pass the Kubernetes Conformance tests. kubeadm also supports other cluster lifecycle functions, such as bootstrap tokens and cluster upgrades.

The kubeadm tool is good if you need:

• A simple way for you to try out Kubernetes, possibly for the first time.
• A way for existing users to automate setting up a cluster and test their application.
• A building block in other ecosystem and/or installer tools with a larger scope.

You can install and use kubeadm on various machines: your laptop, a set of cloud servers, a Raspberry Pi, and more. Whether you're deploying into the cloud or on-premises, you can integrate kubeadm into provisioning systems such as Ansible or Terraform.

Before you begin

To follow this guide, you need:

• One or more machines running a deb/rpm-compatible Linux OS; for example: Ubuntu or CentOS.
• 2 GiB or more of RAM per machine--any less leaves little room for your apps.
• At least 2 CPUs on the machine that you use as a control-plane node.
• Full network connectivity among all machines in the cluster. You can use either a public or a private network.

You also need to use a version of kubeadm that can deploy the version of Kubernetes that you want to use in your new cluster.

Kubernetes' version and version skew support policy applies to kubeadm as well as to Kubernetes overall. Check that policy to learn about what versions of Kubernetes and kubeadm are supported. This page is written for Kubernetes v1.22.

The kubeadm tool's overall feature state is General Availability (GA). Some sub-features are still under active development. The implementation of creating the cluster may change slightly as the tool evolves, but the overall implementation should be pretty stable.

Note: Any commands under kubeadm alpha are, by definition, supported on an alpha level.

Objectives

• Install a single control-plane Kubernetes cluster
• Install a Pod network on the cluster so that your Pods can talk to each other

Instructions

Installing kubeadm on your hosts

See "Installing kubeadm".

Note:

If you have already installed kubeadm, run apt-get update && apt-get upgrade or yum update to get the latest version of kubeadm.

When you upgrade, the kubelet restarts every few seconds as it waits in a crashloop for kubeadm to tell it what to do. This crashloop is expected and normal. After you initialize your control-plane, the kubelet runs normally.

Initializing your control-plane node

The control-plane node is the machine where the control plane components run, including etcd (the cluster database) and the API Server (which the kubectl command line tool communicates with).

1. (Recommended) If you have plans to upgrade this single control-plane kubeadm cluster to high availability you should specify the --control-plane-endpoint to set the shared endpoint for all control-plane nodes. Such an endpoint can be either a DNS name or an IP address of a load-balancer.
2. Choose a Pod network add-on, and verify whether it requires any arguments to be passed to kubeadm init. Depending on which third-party provider you choose, you might need to set the --pod-network-cidr to a provider-specific value. See Installing a Pod network add-on.
3. (Optional) Since version 1.14, kubeadm tries to detect the container runtime on Linux by using a list of well known domain socket paths. To use different container runtime or if there are more than one installed on the provisioned node, specify the --cri-socket argument to kubeadm init. See Installing runtime.
4. (Optional) Unless otherwise specified, kubeadm uses the network interface associated with the default gateway to set the advertise address for this particular control-plane node's API server. To use a different network interface, specify the --apiserver-advertise-address=<ip-address> argument to kubeadm init. To deploy an IPv6 Kubernetes cluster using IPv6 addressing, you must specify an IPv6 address, for example --apiserver-advertise-address=fd00::101
5. (Optional) Run kubeadm config images pull prior to kubeadm init to verify connectivity to the gcr.io container image registry.

To initialize the control-plane node run:

kubeadm init <args>

Considerations about apiserver-advertise-address and ControlPlaneEndpoint

While --apiserver-advertise-address can be used to set the advertise address for this particular control-plane node's API server, --control-plane-endpoint can be used to set the shared endpoint for all control-plane nodes.

--control-plane-endpoint allows both IP addresses and DNS names that can map to IP addresses. Please contact your network administrator to evaluate possible solutions with respect to such mapping.

Here is an example mapping:

192.168.0.102 cluster-endpoint

Where 192.168.0.102 is the IP address of this node and cluster-endpoint is a custom DNS name that maps to this IP. This will allow you to pass --control-plane-endpoint=cluster-endpoint to kubeadm init and pass the same DNS name to kubeadm join. Later you can modify cluster-endpoint to point to the address of your load-balancer in an high availability scenario.

Turning a single control plane cluster created without --control-plane-endpoint into a highly available cluster is not supported by kubeadm.

More information

For more information about kubeadm init arguments, see the kubeadm reference guide.

To configure kubeadm init with a configuration file see Using kubeadm init with a configuration file.

To customize control plane components, including optional IPv6 assignment to liveness probe for control plane components and etcd server, provide extra arguments to each component as documented in custom arguments.

To run kubeadm init again, you must first tear down the cluster.

If you join a node with a different architecture to your cluster, make sure that your deployed DaemonSets have container image support for this architecture.

kubeadm init first runs a series of prechecks to ensure that the machine is ready to run Kubernetes. These prechecks expose warnings and exit on errors. kubeadm init then downloads and installs the cluster control plane components. This may take several minutes. After it finishes you should see:

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a Pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  /docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join <control-plane-host>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>

To make kubectl work for your non-root user, run these commands, which are also part of the kubeadm init output:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

export KUBECONFIG=/etc/kubernetes/admin.conf

Warning: Kubeadm signs the certificate in the admin.conf to have Subject: O = system:masters, CN = kubernetes-admin. system:masters is a break-glass, super user group that bypasses the authorization layer (e.g. RBAC). Do not share the admin.conf file with anyone and instead grant users custom permissions by generating them a kubeconfig file using the kubeadm kubeconfig user command.

Make a record of the kubeadm join command that kubeadm init outputs. You need this command to join nodes to your cluster.

The token is used for mutual authentication between the control-plane node and the joining nodes. The token included here is secret. Keep it safe, because anyone with this token can add authenticated nodes to your cluster. These tokens can be listed, created, and deleted with the kubeadm token command. See the kubeadm reference guide.

Installing a Pod network add-on

Caution:

This section contains important information about networking setup and deployment order. Read all of this advice carefully before proceeding.

You must deploy a Container Network Interface (CNI) based Pod network add-on so that your Pods can communicate with each other. Cluster DNS (CoreDNS) will not start up before a network is installed.

• Take care that your Pod network must not overlap with any of the host networks: you are likely to see problems if there is any overlap. (If you find a collision between your network plugin's preferred Pod network and some of your host networks, you should think of a suitable CIDR block to use instead, then use that during kubeadm init with --pod-network-cidr and as a replacement in your network plugin's YAML).

• By default, kubeadm sets up your cluster to use and enforce use of RBAC (role based access control). Make sure that your Pod network plugin supports RBAC, and so do any manifests that you use to deploy it.

• If you want to use IPv6--either dual-stack, or single-stack IPv6 only networking--for your cluster, make sure that your Pod network plugin supports IPv6. IPv6 support was added to CNI in v0.6.0.

Note: Kubeadm should be CNI agnostic and the validation of CNI providers is out of the scope of our current e2e testing. If you find an issue related to a CNI plugin you should log a ticket in its respective issue tracker instead of the kubeadm or kubernetes issue trackers.

Several external projects provide Kubernetes Pod networks using CNI, some of which also support Network Policy.

See a list of add-ons that implement the Kubernetes networking model.

You can install a Pod network add-on with the following command on the control-plane node or a node that has the kubeconfig credentials:

kubectl apply -f <add-on.yaml>

You can install only one Pod network per cluster.

Once a Pod network has been installed, you can confirm that it is working by checking that the CoreDNS Pod is Running in the output of kubectl get pods --all-namespaces. And once the CoreDNS Pod is up and running, you can continue by joining your nodes.

If your network is not working or CoreDNS is not in the Running state, check out the troubleshooting guide for kubeadm.

Control plane node isolation

By default, your cluster will not schedule Pods on the control-plane node for security reasons. If you want to be able to schedule Pods on the control-plane node, for example for a single-machine Kubernetes cluster for development, run:

kubectl taint nodes --all node-role.kubernetes.io/master-

With output looking something like:

node "test-01" untainted
taint "node-role.kubernetes.io/master:" not found
taint "node-role.kubernetes.io/master:" not found

This will remove the node-role.kubernetes.io/master taint from any nodes that have it, including the control-plane node, meaning that the scheduler will then be able to schedule Pods everywhere.

Joining your nodes

The nodes are where your workloads (containers and Pods, etc) run. To add new nodes to your cluster do the following for each machine:

• SSH to the machine
• Become root (e.g. sudo su -)
• Run the command that was output by kubeadm init. For example:

kubeadm join --token <token> <control-plane-host>:<control-plane-port> --discovery-token-ca-cert-hash sha256:<hash>

If you do not have the token, you can get it by running the following command on the control-plane node:

kubeadm token list

The output is similar to this:

TOKEN                    TTL  EXPIRES              USAGES           DESCRIPTION            EXTRA GROUPS
8ewj1p.9r9hcjoqgajrj4gi  23h  2018-06-12T02:51:28Z authentication,  The default bootstrap  system:
                                                   signing          token generated by     bootstrappers:
                                                                    'kubeadm init'.        kubeadm:
                                                                                           default-node-token

By default, tokens expire after 24 hours. If you are joining a node to the cluster after the current token has expired, you can create a new token by running the following command on the control-plane node:

kubeadm token create

The output is similar to this:

5didvk.d09sbcov8ph2amjw

If you don't have the value of --discovery-token-ca-cert-hash, you can get it by running the following command chain on the control-plane node:

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
   openssl dgst -sha256 -hex | sed 's/^.* //'

The output is similar to:

8cb2de97839780a412b93877f8507ad6c94f73add17d5d7058e91741c9d5ec78

Note: To specify an IPv6 tuple for <control-plane-host>:<control-plane-port>, IPv6 address must be enclosed in square brackets, for example: [fd00::101]:2073.

The output should look something like:

[preflight] Running pre-flight checks

... (log output of join workflow) ...

Node join complete:
* Certificate signing request sent to control-plane and response
  received.
* Kubelet informed of new secure connection details.

Run 'kubectl get nodes' on control-plane to see this machine join.

A few seconds later, you should notice this node in the output from kubectl get nodes when run on the control-plane node.

(Optional) Controlling your cluster from machines other than the control-plane node

In order to get a kubectl on some other computer (e.g. laptop) to talk to your cluster, you need to copy the administrator kubeconfig file from your control-plane node to your workstation like this:

scp root@<control-plane-host>:/etc/kubernetes/admin.conf .
kubectl --kubeconfig ./admin.conf get nodes

Note:

The example above assumes SSH access is enabled for root. If that is not the case, you can copy the admin.conf file to be accessible by some other user and scp using that other user instead.

The admin.conf file gives the user superuser privileges over the cluster. This file should be used sparingly. For normal users, it's recommended to generate an unique credential to which you grant privileges. You can do this with the kubeadm alpha kubeconfig user --client-name <CN> command. That command will print out a KubeConfig file to STDOUT which you should save to a file and distribute to your user. After that, grant privileges by using kubectl create (cluster)rolebinding.

(Optional) Proxying API Server to localhost

If you want to connect to the API Server from outside the cluster you can use kubectl proxy:

scp root@<control-plane-host>:/etc/kubernetes/admin.conf .
kubectl --kubeconfig ./admin.conf proxy

You can now access the API Server locally at http://localhost:8001/api/v1

Clean up

If you used disposable servers for your cluster, for testing, you can switch those off and do no further clean up. You can use kubectl config delete-cluster to delete your local references to the cluster.

However, if you want to deprovision your cluster more cleanly, you should first drain the node and make sure that the node is empty, then deconfigure the node.

Remove the node

Talking to the control-plane node with the appropriate credentials, run:

kubectl drain <node name> --delete-local-data --force --ignore-daemonsets

Before removing the node, reset the state installed by kubeadm:

kubeadm reset

The reset process does not reset or clean up iptables rules or IPVS tables. If you wish to reset iptables, you must do so manually:

iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

If you want to reset the IPVS tables, you must run the following command:

ipvsadm -C

Now remove the node:

kubectl delete node <node name>

If you wish to start over, run kubeadm init or kubeadm join with the appropriate arguments.

Clean up the control plane

You can use kubeadm reset on the control plane host to trigger a best-effort clean up.

See the kubeadm reset reference documentation for more information about this subcommand and its options.

What's next

• Verify that your cluster is running properly with Sonobuoy
• See Upgrading kubeadm clusters for details about upgrading your cluster using kubeadm.
• Learn about advanced kubeadm usage in the kubeadm reference documentation
• Learn more about Kubernetes concepts and kubectl.
• See the Cluster Networking page for a bigger list of Pod network add-ons.
• See the list of add-ons to explore other add-ons, including tools for logging, monitoring, network policy, visualization & control of your Kubernetes cluster.
• Configure how your cluster handles logs for cluster events and from applications running in Pods. See Logging Architecture for an overview of what is involved.

Feedback

• For bugs, visit the kubeadm GitHub issue tracker
• For support, visit the #kubeadm Slack channel
• General SIG Cluster Lifecycle development Slack channel: #sig-cluster-lifecycle
• SIG Cluster Lifecycle SIG information
• SIG Cluster Lifecycle mailing list: kubernetes-sig-cluster-lifecycle

Version skew policy

The kubeadm tool of version v1.22 may deploy clusters with a control plane of version v1.22 or v1.21. kubeadm v1.22 can also upgrade an existing kubeadm-created cluster of version v1.21.

Due to that we can't see into the future, kubeadm CLI v1.22 may or may not be able to deploy v1.23 clusters.

These resources provide more information on supported version skew between kubelets and the control plane, and other Kubernetes components:

• Kubernetes version and version-skew policy
• Kubeadm-specific installation guide

Limitations

Cluster resilience

The cluster created here has a single control-plane node, with a single etcd database running on it. This means that if the control-plane node fails, your cluster may lose data and may need to be recreated from scratch.

Workarounds:

• Regularly back up etcd. The etcd data directory configured by kubeadm is at /var/lib/etcd on the control-plane node.

• Use multiple control-plane nodes. You can read Options for Highly Available topology to pick a cluster topology that provides high-availability.

Platform compatibility

kubeadm deb/rpm packages and binaries are built for amd64, arm (32-bit), arm64, ppc64le, and s390x following the multi-platform proposal.

Multiplatform container images for the control plane and addons are also supported since v1.12.

Only some of the network providers offer solutions for all platforms. Please consult the list of network providers above or the documentation from each provider to figure out whether the provider supports your chosen platform.

Troubleshooting

If you are running into difficulties with kubeadm, please consult our troubleshooting docs.

3.2.1.4 - Customizing control plane configuration with kubeadm

The kubeadm ClusterConfiguration object exposes the field extraArgs that can override the default flags passed to control plane components such as the APIServer, ControllerManager and Scheduler. The components are defined using the following fields:

• apiServer
• controllerManager
• scheduler

The extraArgs field consist of key: value pairs. To override a flag for a control plane component:

1. Add the appropriate fields to your configuration.
2. Add the flags to override to the field.
3. Run kubeadm init with --config <YOUR CONFIG YAML>.

For more details on each field in the configuration you can navigate to our API reference pages.

Note: You can generate a ClusterConfiguration object with default values by running kubeadm config print init-defaults and saving the output to a file of your choice.

APIServer flags

For details, see the reference documentation for kube-apiserver.

Example usage:

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.16.0
apiServer:
  extraArgs:
    advertise-address: 192.168.0.103
    anonymous-auth: "false"
    enable-admission-plugins: AlwaysPullImages,DefaultStorageClass
    audit-log-path: /home/johndoe/audit.log

ControllerManager flags

For details, see the reference documentation for kube-controller-manager.

Example usage:

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.16.0
controllerManager:
  extraArgs:
    cluster-signing-key-file: /home/johndoe/keys/ca.key
    bind-address: 0.0.0.0
    deployment-controller-sync-period: "50"

Scheduler flags

For details, see the reference documentation for kube-scheduler.

Example usage:

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.16.0
scheduler:
  extraArgs:
    bind-address: 0.0.0.0
    config: /home/johndoe/schedconfig.yaml
    kubeconfig: /home/johndoe/kubeconfig.yaml

3.2.1.5 - Options for Highly Available topology

This page explains the two options for configuring the topology of your highly available (HA) Kubernetes clusters.

You can set up an HA cluster:

• With stacked control plane nodes, where etcd nodes are colocated with control plane nodes
• With external etcd nodes, where etcd runs on separate nodes from the control plane

You should carefully consider the advantages and disadvantages of each topology before setting up an HA cluster.

Note: kubeadm bootstraps the etcd cluster statically. Read the etcd Clustering Guide for more details.

Stacked etcd topology

A stacked HA cluster is a topology where the distributed data storage cluster provided by etcd is stacked on top of the cluster formed by the nodes managed by kubeadm that run control plane components.

Each control plane node runs an instance of the kube-apiserver, kube-scheduler, and kube-controller-manager. The kube-apiserver is exposed to worker nodes using a load balancer.

Each control plane node creates a local etcd member and this etcd member communicates only with the kube-apiserver of this node. The same applies to the local kube-controller-manager and kube-scheduler instances.

This topology couples the control planes and etcd members on the same nodes. It is simpler to set up than a cluster with external etcd nodes, and simpler to manage for replication.

However, a stacked cluster runs the risk of failed coupling. If one node goes down, both an etcd member and a control plane instance are lost, and redundancy is compromised. You can mitigate this risk by adding more control plane nodes.

You should therefore run a minimum of three stacked control plane nodes for an HA cluster.

This is the default topology in kubeadm. A local etcd member is created automatically on control plane nodes when using kubeadm init and kubeadm join --control-plane.

External etcd topology

An HA cluster with external etcd is a topology where the distributed data storage cluster provided by etcd is external to the cluster formed by the nodes that run control plane components.

Like the stacked etcd topology, each control plane node in an external etcd topology runs an instance of the kube-apiserver, kube-scheduler, and kube-controller-manager. And the kube-apiserver is exposed to worker nodes using a load balancer. However, etcd members run on separate hosts, and each etcd host communicates with the kube-apiserver of each control plane node.

This topology decouples the control plane and etcd member. It therefore provides an HA setup where losing a control plane instance or an etcd member has less impact and does not affect the cluster redundancy as much as the stacked HA topology.

However, this topology requires twice the number of hosts as the stacked HA topology. A minimum of three hosts for control plane nodes and three hosts for etcd nodes are required for an HA cluster with this topology.

What's next

• Set up a highly available cluster with kubeadm

3.2.1.6 - Creating Highly Available clusters with kubeadm

This page explains two different approaches to setting up a highly available Kubernetes cluster using kubeadm:

• With stacked control plane nodes. This approach requires less infrastructure. The etcd members and control plane nodes are co-located.
• With an external etcd cluster. This approach requires more infrastructure. The control plane nodes and etcd members are separated.

Before proceeding, you should carefully consider which approach best meets the needs of your applications and environment. This comparison topic outlines the advantages and disadvantages of each.

If you encounter issues with setting up the HA cluster, please provide us with feedback in the kubeadm issue tracker.

See also The upgrade documentation.

Caution: This page does not address running your cluster on a cloud provider. In a cloud environment, neither approach documented here works with Service objects of type LoadBalancer, or with dynamic PersistentVolumes.

Before you begin

For both methods you need this infrastructure:

• Three machines that meet kubeadm's minimum requirements for the control-plane nodes
• Three machines that meet kubeadm's minimum requirements for the workers
• Full network connectivity between all machines in the cluster (public or private network)
• sudo privileges on all machines
• SSH access from one device to all nodes in the system
• kubeadm and kubelet installed on all machines. kubectl is optional.

For the external etcd cluster only, you also need:

• Three additional machines for etcd members

First steps for both methods

Create load balancer for kube-apiserver

Note: There are many configurations for load balancers. The following example is only one option. Your cluster requirements may need a different configuration.

1. Create a kube-apiserver load balancer with a name that resolves to DNS.

   • In a cloud environment you should place your control plane nodes behind a TCP forwarding load balancer. This load balancer distributes traffic to all healthy control plane nodes in its target list. The health check for an apiserver is a TCP check on the port the kube-apiserver listens on (default value :6443).

   • It is not recommended to use an IP address directly in a cloud environment.

   • The load balancer must be able to communicate with all control plane nodes on the apiserver port. It must also allow incoming traffic on its listening port.

   • Make sure the address of the load balancer always matches the address of kubeadm's ControlPlaneEndpoint.

   • Read the Options for Software Load Balancing guide for more details.

2. Add the first control plane nodes to the load balancer and test the connection:

   nc -v LOAD_BALANCER_IP PORT
   

   • A connection refused error is expected because the apiserver is not yet running. A timeout, however, means the load balancer cannot communicate with the control plane node. If a timeout occurs, reconfigure the load balancer to communicate with the control plane node.

3. Add the remaining control plane nodes to the load balancer target group.

Stacked control plane and etcd nodes

Steps for the first control plane node

1. Initialize the control plane:

   sudo kubeadm init --control-plane-endpoint "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT" --upload-certs
   

   • You can use the --kubernetes-version flag to set the Kubernetes version to use. It is recommended that the versions of kubeadm, kubelet, kubectl and Kubernetes match.

   • The --control-plane-endpoint flag should be set to the address or DNS and port of the load balancer.

   • The --upload-certs flag is used to upload the certificates that should be shared across all the control-plane instances to the cluster. If instead, you prefer to copy certs across control-plane nodes manually or using automation tools, please remove this flag and refer to Manual certificate distribution section below.

   Note: The kubeadm init flags --config and --certificate-key cannot be mixed, therefore if you want to use the kubeadm configuration you must add the certificateKey field in the appropriate config locations (under InitConfiguration and JoinConfiguration: controlPlane).

   Note: Some CNI network plugins require additional configuration, for example specifying the pod IP CIDR, while others do not. See the CNI network documentation. To add a pod CIDR pass the flag --pod-network-cidr, or if you are using a kubeadm configuration file set the podSubnet field under the networking object of ClusterConfiguration.

   • The output looks similar to:

     ...
     You can now join any number of control-plane node by running the following command on each as a root:
         kubeadm join 192.168.0.200:6443 --token 9vr73a.a8uxyaju799qwdjv --discovery-token-ca-cert-hash sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866 --control-plane --certificate-key f8902e114ef118304e561c3ecd4d0b543adc226b7a07f675f56564185ffe0c07
     
     Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
     As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use kubeadm init phase upload-certs to reload certs afterward.
     
     Then you can join any number of worker nodes by running the following on each as root:
         kubeadm join 192.168.0.200:6443 --token 9vr73a.a8uxyaju799qwdjv --discovery-token-ca-cert-hash sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866
     

   • Copy this output to a text file. You will need it later to join control plane and worker nodes to the cluster.

   • When --upload-certs is used with kubeadm init, the certificates of the primary control plane are encrypted and uploaded in the kubeadm-certs Secret.

   • To re-upload the certificates and generate a new decryption key, use the following command on a control plane node that is already joined to the cluster:

     sudo kubeadm init phase upload-certs --upload-certs
     

   • You can also specify a custom --certificate-key during init that can later be used by join. To generate such a key you can use the following command:

     kubeadm certs certificate-key
     

   Note: The kubeadm-certs Secret and decryption key expire after two hours.

   Caution: As stated in the command output, the certificate key gives access to cluster sensitive data, keep it secret!

2. Apply the CNI plugin of your choice: Follow these instructions to install the CNI provider. Make sure the configuration corresponds to the Pod CIDR specified in the kubeadm configuration file if applicable.

   In this example we are using Weave Net:

   kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
   

3. Type the following and watch the pods of the control plane components get started:

   kubectl get pod -n kube-system -w
   

Steps for the rest of the control plane nodes

Note: Since kubeadm version 1.15 you can join multiple control-plane nodes in parallel. Prior to this version, you must join new control plane nodes sequentially, only after the first node has finished initializing.

For each additional control plane node you should:

1. Execute the join command that was previously given to you by the kubeadm init output on the first node. It should look something like this:

   sudo kubeadm join 192.168.0.200:6443 --token 9vr73a.a8uxyaju799qwdjv --discovery-token-ca-cert-hash sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866 --control-plane --certificate-key f8902e114ef118304e561c3ecd4d0b543adc226b7a07f675f56564185ffe0c07
   

   • The --control-plane flag tells kubeadm join to create a new control plane.
   • The --certificate-key ... will cause the control plane certificates to be downloaded from the kubeadm-certs Secret in the cluster and be decrypted using the given key.

External etcd nodes

Setting up a cluster with external etcd nodes is similar to the procedure used for stacked etcd with the exception that you should setup etcd first, and you should pass the etcd information in the kubeadm config file.

Set up the etcd cluster

1. Follow these instructions to set up the etcd cluster.

2. Setup SSH as described here.

3. Copy the following files from any etcd node in the cluster to the first control plane node:

   export CONTROL_PLANE="ubuntu@10.0.0.7"
   scp /etc/kubernetes/pki/etcd/ca.crt "${CONTROL_PLANE}":
   scp /etc/kubernetes/pki/apiserver-etcd-client.crt "${CONTROL_PLANE}":
   scp /etc/kubernetes/pki/apiserver-etcd-client.key "${CONTROL_PLANE}":
   

   • Replace the value of CONTROL_PLANE with the user@host of the first control-plane node.

Set up the first control plane node

1. Create a file called kubeadm-config.yaml with the following contents:

   apiVersion: kubeadm.k8s.io/v1beta2
   kind: ClusterConfiguration
   kubernetesVersion: stable
   controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT"
   etcd:
       external:
           endpoints:
           - https://ETCD_0_IP:2379
           - https://ETCD_1_IP:2379
           - https://ETCD_2_IP:2379
           caFile: /etc/kubernetes/pki/etcd/ca.crt
           certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
           keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
   

Note: The difference between stacked etcd and external etcd here is that the external etcd setup requires a configuration file with the etcd endpoints under the external object for etcd. In the case of the stacked etcd topology this is managed automatically.

• Replace the following variables in the config template with the appropriate values for your cluster:

- `LOAD_BALANCER_DNS`
- `LOAD_BALANCER_PORT`
- `ETCD_0_IP`
- `ETCD_1_IP`
- `ETCD_2_IP`

The following steps are similar to the stacked etcd setup:

1. Run sudo kubeadm init --config kubeadm-config.yaml --upload-certs on this node.

2. Write the output join commands that are returned to a text file for later use.

3. Apply the CNI plugin of your choice. The given example is for Weave Net:

   kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
   

Steps for the rest of the control plane nodes

The steps are the same as for the stacked etcd setup:

• Make sure the first control plane node is fully initialized.
• Join each control plane node with the join command you saved to a text file. It's recommended to join the control plane nodes one at a time.
• Don't forget that the decryption key from --certificate-key expires after two hours, by default.

Common tasks after bootstrapping control plane

Install workers

Worker nodes can be joined to the cluster with the command you stored previously as the output from the kubeadm init command:

sudo kubeadm join 192.168.0.200:6443 --token 9vr73a.a8uxyaju799qwdjv --discovery-token-ca-cert-hash sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866

Manual certificate distribution

If you choose to not use kubeadm init with the --upload-certs flag this means that you are going to have to manually copy the certificates from the primary control plane node to the joining control plane nodes.

There are many ways to do this. In the following example we are using ssh and scp:

SSH is required if you want to control all nodes from a single machine.

1. Enable ssh-agent on your main device that has access to all other nodes in the system:

   eval $(ssh-agent)
   

2. Add your SSH identity to the session:

   ssh-add ~/.ssh/path_to_private_key
   

3. SSH between nodes to check that the connection is working correctly.

   • When you SSH to any node, make sure to add the -A flag:

     ssh -A 10.0.0.7
     

   • When using sudo on any node, make sure to preserve the environment so SSH forwarding works:

     sudo -E -s
     

4. After configuring SSH on all the nodes you should run the following script on the first control plane node after running kubeadm init. This script will copy the certificates from the first control plane node to the other control plane nodes:

   In the following example, replace CONTROL_PLANE_IPS with the IP addresses of the other control plane nodes.

   USER=ubuntu # customizable
   CONTROL_PLANE_IPS="10.0.0.7 10.0.0.8"
   for host in ${CONTROL_PLANE_IPS}; do
       scp /etc/kubernetes/pki/ca.crt "${USER}"@$host:
       scp /etc/kubernetes/pki/ca.key "${USER}"@$host:
       scp /etc/kubernetes/pki/sa.key "${USER}"@$host:
       scp /etc/kubernetes/pki/sa.pub "${USER}"@$host:
       scp /etc/kubernetes/pki/front-proxy-ca.crt "${USER}"@$host:
       scp /etc/kubernetes/pki/front-proxy-ca.key "${USER}"@$host:
       scp /etc/kubernetes/pki/etcd/ca.crt "${USER}"@$host:etcd-ca.crt
       # Quote this line if you are using external etcd
       scp /etc/kubernetes/pki/etcd/ca.key "${USER}"@$host:etcd-ca.key
   done
   

   Caution: Copy only the certificates in the above list. kubeadm will take care of generating the rest of the certificates with the required SANs for the joining control-plane instances. If you copy all the certificates by mistake, the creation of additional nodes could fail due to a lack of required SANs.

5. Then on each joining control plane node you have to run the following script before running kubeadm join. This script will move the previously copied certificates from the home directory to /etc/kubernetes/pki:

   USER=ubuntu # customizable
   mkdir -p /etc/kubernetes/pki/etcd
   mv /home/${USER}/ca.crt /etc/kubernetes/pki/
   mv /home/${USER}/ca.key /etc/kubernetes/pki/
   mv /home/${USER}/sa.pub /etc/kubernetes/pki/
   mv /home/${USER}/sa.key /etc/kubernetes/pki/
   mv /home/${USER}/front-proxy-ca.crt /etc/kubernetes/pki/
   mv /home/${USER}/front-proxy-ca.key /etc/kubernetes/pki/
   mv /home/${USER}/etcd-ca.crt /etc/kubernetes/pki/etcd/ca.crt
   # Quote this line if you are using external etcd
   mv /home/${USER}/etcd-ca.key /etc/kubernetes/pki/etcd/ca.key
   

3.2.1.7 - Set up a High Availability etcd cluster with kubeadm

Note: While kubeadm is being used as the management tool for external etcd nodes in this guide, please note that kubeadm does not plan to support certificate rotation or upgrades for such nodes. The long term plan is to empower the tool etcdadm to manage these aspects.

Kubeadm defaults to running a single member etcd cluster in a static pod managed by the kubelet on the control plane node. This is not a high availability setup as the etcd cluster contains only one member and cannot sustain any members becoming unavailable. This task walks through the process of creating a high availability etcd cluster of three members that can be used as an external etcd when using kubeadm to set up a kubernetes cluster.

Before you begin

• Three hosts that can talk to each other over ports 2379 and 2380. This document assumes these default ports. However, they are configurable through the kubeadm config file.
• Each host must have docker, kubelet, and kubeadm installed.
• Each host should have access to the Kubernetes container image registry (k8s.gcr.io) or list/pull the required etcd image using kubeadm config images list/pull. This guide will setup etcd instances as static pods managed by a kubelet.
• Some infrastructure to copy files between hosts. For example ssh and scp can satisfy this requirement.

Setting up the cluster

The general approach is to generate all certs on one node and only distribute the necessary files to the other nodes.

Note: kubeadm contains all the necessary crytographic machinery to generate the certificates described below; no other cryptographic tooling is required for this example.

1. Configure the kubelet to be a service manager for etcd.

   Note: You must do this on every host where etcd should be running.

   Since etcd was created first, you must override the service priority by creating a new unit file that has higher precedence than the kubeadm-provided kubelet unit file.

   cat << EOF > /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf
   [Service]
   ExecStart=
   #  Replace "systemd" with the cgroup driver of your container runtime. The default value in the kubelet is "cgroupfs".
   ExecStart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --cgroup-driver=systemd
   Restart=always
   EOF
   
   systemctl daemon-reload
   systemctl restart kubelet
   

   Check the kubelet status to ensure it is running.

   systemctl status kubelet
   

2. Create configuration files for kubeadm.

   Generate one kubeadm configuration file for each host that will have an etcd member running on it using the following script.

   # Update HOST0, HOST1, and HOST2 with the IPs or resolvable names of your hosts
   export HOST0=10.0.0.6
   export HOST1=10.0.0.7
   export HOST2=10.0.0.8
   
   # Create temp directories to store files that will end up on other hosts.
   mkdir -p /tmp/${HOST0}/ /tmp/${HOST1}/ /tmp/${HOST2}/
   
   ETCDHOSTS=(${HOST0} ${HOST1} ${HOST2})
   NAMES=("infra0" "infra1" "infra2")
   
   for i in "${!ETCDHOSTS[@]}"; do
   HOST=${ETCDHOSTS[$i]}
   NAME=${NAMES[$i]}
   cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml
   apiVersion: "kubeadm.k8s.io/v1beta2"
   kind: ClusterConfiguration
   etcd:
       local:
           serverCertSANs:
           - "${HOST}"
           peerCertSANs:
           - "${HOST}"
           extraArgs:
               initial-cluster: ${NAMES[0]}=https://${ETCDHOSTS[0]}:2380,${NAMES[1]}=https://${ETCDHOSTS[1]}:2380,${NAMES[2]}=https://${ETCDHOSTS[2]}:2380
               initial-cluster-state: new
               name: ${NAME}
               listen-peer-urls: https://${HOST}:2380
               listen-client-urls: https://${HOST}:2379
               advertise-client-urls: https://${HOST}:2379
               initial-advertise-peer-urls: https://${HOST}:2380
   EOF
   done
   

3. Generate the certificate authority

   If you already have a CA then the only action that is copying the CA's crt and key file to /etc/kubernetes/pki/etcd/ca.crt and /etc/kubernetes/pki/etcd/ca.key. After those files have been copied, proceed to the next step, "Create certificates for each member".

   If you do not already have a CA then run this command on $HOST0 (where you generated the configuration files for kubeadm).

   kubeadm init phase certs etcd-ca
   

   This creates two files

   • /etc/kubernetes/pki/etcd/ca.crt
   • /etc/kubernetes/pki/etcd/ca.key

4. Create certificates for each member

   kubeadm init phase certs etcd-server --config=/tmp/${HOST2}/kubeadmcfg.yaml
   kubeadm init phase certs etcd-peer --config=/tmp/${HOST2}/kubeadmcfg.yaml
   kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
   kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
   cp -R /etc/kubernetes/pki /tmp/${HOST2}/
   # cleanup non-reusable certificates
   find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete
   
   kubeadm init phase certs etcd-server --config=/tmp/${HOST1}/kubeadmcfg.yaml
   kubeadm init phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml
   kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
   kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
   cp -R /etc/kubernetes/pki /tmp/${HOST1}/
   find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete
   
   kubeadm init phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml
   kubeadm init phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml
   kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
   kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
   # No need to move the certs because they are for HOST0
   
   # clean up certs that should not be copied off this host
   find /tmp/${HOST2} -name ca.key -type f -delete
   find /tmp/${HOST1} -name ca.key -type f -delete
   

5. Copy certificates and kubeadm configs

   The certificates have been generated and now they must be moved to their respective hosts.

   USER=ubuntu
   HOST=${HOST1}
   scp -r /tmp/${HOST}/* ${USER}@${HOST}:
   ssh ${USER}@${HOST}
   USER@HOST $ sudo -Es
   root@HOST $ chown -R root:root pki
   root@HOST $ mv pki /etc/kubernetes/
   

6. Ensure all expected files exist

   The complete list of required files on $HOST0 is:

   /tmp/${HOST0}
   └── kubeadmcfg.yaml
   ---
   /etc/kubernetes/pki
   ├── apiserver-etcd-client.crt
   ├── apiserver-etcd-client.key
   └── etcd
       ├── ca.crt
       ├── ca.key
       ├── healthcheck-client.crt
       ├── healthcheck-client.key
       ├── peer.crt
       ├── peer.key
       ├── server.crt
       └── server.key
   

   On $HOST1:

   $HOME
   └── kubeadmcfg.yaml
   ---
   /etc/kubernetes/pki
   ├── apiserver-etcd-client.crt
   ├── apiserver-etcd-client.key
   └── etcd
       ├── ca.crt
       ├── healthcheck-client.crt
       ├── healthcheck-client.key
       ├── peer.crt
       ├── peer.key
       ├── server.crt
       └── server.key
   

   On $HOST2

   $HOME
   └── kubeadmcfg.yaml
   ---
   /etc/kubernetes/pki
   ├── apiserver-etcd-client.crt
   ├── apiserver-etcd-client.key
   └── etcd
       ├── ca.crt
       ├── healthcheck-client.crt
       ├── healthcheck-client.key
       ├── peer.crt
       ├── peer.key
       ├── server.crt
       └── server.key
   

7. Create the static pod manifests

   Now that the certificates and configs are in place it's time to create the manifests. On each host run the kubeadm command to generate a static manifest for etcd.

   root@HOST0 $ kubeadm init phase etcd local --config=/tmp/${HOST0}/kubeadmcfg.yaml
   root@HOST1 $ kubeadm init phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml
   root@HOST2 $ kubeadm init phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml
   

8. Optional: Check the cluster health

   docker run --rm -it \
   --net host \
   -v /etc/kubernetes:/etc/kubernetes k8s.gcr.io/etcd:${ETCD_TAG} etcdctl \
   --cert /etc/kubernetes/pki/etcd/peer.crt \
   --key /etc/kubernetes/pki/etcd/peer.key \
   --cacert /etc/kubernetes/pki/etcd/ca.crt \
   --endpoints https://${HOST0}:2379 endpoint health --cluster
   ...
   https://[HOST0 IP]:2379 is healthy: successfully committed proposal: took = 16.283339ms
   https://[HOST1 IP]:2379 is healthy: successfully committed proposal: took = 19.44402ms
   https://[HOST2 IP]:2379 is healthy: successfully committed proposal: took = 35.926451ms
   

   • Set ${ETCD_TAG} to the version tag of your etcd image. For example 3.4.3-0. To see the etcd image and tag that kubeadm uses execute kubeadm config images list --kubernetes-version ${K8S_VERSION}, where ${K8S_VERSION} is for example v1.17.0
   • Set ${HOST0}to the IP address of the host you are testing.

What's next

Once you have a working 3 member etcd cluster, you can continue setting up a highly available control plane using the external etcd method with kubeadm.

3.2.1.8 - Configuring each kubelet in your cluster using kubeadm

The lifecycle of the kubeadm CLI tool is decoupled from the kubelet, which is a daemon that runs on each node within the Kubernetes cluster. The kubeadm CLI tool is executed by the user when Kubernetes is initialized or upgraded, whereas the kubelet is always running in the background.

Since the kubelet is a daemon, it needs to be maintained by some kind of an init system or service manager. When the kubelet is installed using DEBs or RPMs, systemd is configured to manage the kubelet. You can use a different service manager instead, but you need to configure it manually.

Some kubelet configuration details need to be the same across all kubelets involved in the cluster, while other configuration aspects need to be set on a per-kubelet basis to accommodate the different characteristics of a given machine (such as OS, storage, and networking). You can manage the configuration of your kubelets manually, but kubeadm now provides a KubeletConfiguration API type for managing your kubelet configurations centrally.

Kubelet configuration patterns

The following sections describe patterns to kubelet configuration that are simplified by using kubeadm, rather than managing the kubelet configuration for each Node manually.

Propagating cluster-level configuration to each kubelet

You can provide the kubelet with default values to be used by kubeadm init and kubeadm join commands. Interesting examples include using a different CRI runtime or setting the default subnet used by services.

If you want your services to use the subnet 10.96.0.0/12 as the default for services, you can pass the --service-cidr parameter to kubeadm:

kubeadm init --service-cidr 10.96.0.0/12

Virtual IPs for services are now allocated from this subnet. You also need to set the DNS address used by the kubelet, using the --cluster-dns flag. This setting needs to be the same for every kubelet on every manager and Node in the cluster. The kubelet provides a versioned, structured API object that can configure most parameters in the kubelet and push out this configuration to each running kubelet in the cluster. This object is called KubeletConfiguration. The KubeletConfiguration allows the user to specify flags such as the cluster DNS IP addresses expressed as a list of values to a camelCased key, illustrated by the following example:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
clusterDNS:
- 10.96.0.10

For more details on the KubeletConfiguration have a look at this section.

Providing instance-specific configuration details

Some hosts require specific kubelet configurations due to differences in hardware, operating system, networking, or other host-specific parameters. The following list provides a few examples.

• The path to the DNS resolution file, as specified by the --resolv-conf kubelet configuration flag, may differ among operating systems, or depending on whether you are using systemd-resolved. If this path is wrong, DNS resolution will fail on the Node whose kubelet is configured incorrectly.

• The Node API object .metadata.name is set to the machine's hostname by default, unless you are using a cloud provider. You can use the --hostname-override flag to override the default behavior if you need to specify a Node name different from the machine's hostname.

• Currently, the kubelet cannot automatically detect the cgroup driver used by the CRI runtime, but the value of --cgroup-driver must match the cgroup driver used by the CRI runtime to ensure the health of the kubelet.

• Depending on the CRI runtime your cluster uses, you may need to specify different flags to the kubelet. For instance, when using Docker, you need to specify flags such as --network-plugin=cni, but if you are using an external runtime, you need to specify --container-runtime=remote and specify the CRI endpoint using the --container-runtime-endpoint=<path>.

You can specify these flags by configuring an individual kubelet's configuration in your service manager, such as systemd.

Configure kubelets using kubeadm

It is possible to configure the kubelet that kubeadm will start if a custom KubeletConfiguration API object is passed with a configuration file like so kubeadm ... --config some-config-file.yaml.

By calling kubeadm config print init-defaults --component-configs KubeletConfiguration you can see all the default values for this structure.

Also have a look at the reference for the KubeletConfiguration for more information on the individual fields.

Workflow when using kubeadm init

When you call kubeadm init, the kubelet configuration is marshalled to disk at /var/lib/kubelet/config.yaml, and also uploaded to a ConfigMap in the cluster. The ConfigMap is named kubelet-config-1.X, where X is the minor version of the Kubernetes version you are initializing. A kubelet configuration file is also written to /etc/kubernetes/kubelet.conf with the baseline cluster-wide configuration for all kubelets in the cluster. This configuration file points to the client certificates that allow the kubelet to communicate with the API server. This addresses the need to propagate cluster-level configuration to each kubelet.

To address the second pattern of providing instance-specific configuration details, kubeadm writes an environment file to /var/lib/kubelet/kubeadm-flags.env, which contains a list of flags to pass to the kubelet when it starts. The flags are presented in the file like this:

KUBELET_KUBEADM_ARGS="--flag1=value1 --flag2=value2 ..."

In addition to the flags used when starting the kubelet, the file also contains dynamic parameters such as the cgroup driver and whether to use a different CRI runtime socket (--cri-socket).

After marshalling these two files to disk, kubeadm attempts to run the following two commands, if you are using systemd:

systemctl daemon-reload && systemctl restart kubelet

If the reload and restart are successful, the normal kubeadm init workflow continues.

Workflow when using kubeadm join

When you run kubeadm join, kubeadm uses the Bootstrap Token credential to perform a TLS bootstrap, which fetches the credential needed to download the kubelet-config-1.X ConfigMap and writes it to /var/lib/kubelet/config.yaml. The dynamic environment file is generated in exactly the same way as kubeadm init.

Next, kubeadm runs the following two commands to load the new configuration into the kubelet:

systemctl daemon-reload && systemctl restart kubelet

After the kubelet loads the new configuration, kubeadm writes the /etc/kubernetes/bootstrap-kubelet.conf KubeConfig file, which contains a CA certificate and Bootstrap Token. These are used by the kubelet to perform the TLS Bootstrap and obtain a unique credential, which is stored in /etc/kubernetes/kubelet.conf. When this file is written, the kubelet has finished performing the TLS Bootstrap.

The kubelet drop-in file for systemd

kubeadm ships with configuration for how systemd should run the kubelet. Note that the kubeadm CLI command never touches this drop-in file.

This configuration file installed by the kubeadm DEB or RPM package is written to /etc/systemd/system/kubelet.service.d/10-kubeadm.conf and is used by systemd. It augments the basic kubelet.service for RPM or kubelet.service for DEB:

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf
--kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generate at runtime, populating
the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably,
# the user should use the .NodeRegistration.KubeletExtraArgs object in the configuration files instead.
# KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

This file specifies the default locations for all of the files managed by kubeadm for the kubelet.

• The KubeConfig file to use for the TLS Bootstrap is /etc/kubernetes/bootstrap-kubelet.conf, but it is only used if /etc/kubernetes/kubelet.conf does not exist.
• The KubeConfig file with the unique kubelet identity is /etc/kubernetes/kubelet.conf.
• The file containing the kubelet's ComponentConfig is /var/lib/kubelet/config.yaml.
• The dynamic environment file that contains KUBELET_KUBEADM_ARGS is sourced from /var/lib/kubelet/kubeadm-flags.env.
• The file that can contain user-specified flag overrides with KUBELET_EXTRA_ARGS is sourced from /etc/default/kubelet (for DEBs), or /etc/sysconfig/kubelet (for RPMs). KUBELET_EXTRA_ARGS is last in the flag chain and has the highest priority in the event of conflicting settings.

Kubernetes binaries and package contents

The DEB and RPM packages shipped with the Kubernetes releases are:

3.2.1.9 - Dual-stack support with kubeadm

Your Kubernetes cluster can run in dual-stack networking mode, which means that cluster networking lets you use either address family. In a dual-stack cluster, the control plane can assign both an IPv4 address and an IPv6 address to a single Pod or a Service.

Before you begin

You need to have installed the kubeadm tool, following the steps from Installing kubeadm.

For each server that you want to use as a node, make sure it allows IPv6 forwarding. On Linux, you can set this by running run sysctl -w net.ipv6.conf.all.forwarding=1 as the root user on each server.

You need to have an IPv4 and and IPv6 address range to use. Cluster operators typically use private address ranges for IPv4. For IPv6, a cluster operator typically chooses a global unicast address block from within 2000::/3, using a range that is assigned to the operator.
You don't have to route the cluster's IP address ranges to the public internet.

The size of the IP address allocations should be suitable for the number of Pods and Services that you are planning to run.

Note: If you are upgrading an existing cluster then, by default, the kubeadm upgrade command changes the feature gate IPv6DualStack to true if that is not already enabled.
However, kubeadm does not support making modifications to the pod IP address range (“cluster CIDR”) nor to the cluster's Service address range (“Service CIDR”).

Create a dual-stack cluster

To create a dual-stack cluster with kubeadm init you can pass command line arguments similar to the following example:

# These address ranges are examples
kubeadm init --pod-network-cidr=10.244.0.0/16,2001:db8:42:0::/56 --service-cidr=10.96.0.0/16,2001:db8:42:1::/112

To make things clearer, here is an example kubeadm configuration file kubeadm-config.yaml for the primary dual-stack control plane node.

---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
featureGates:
  IPv6DualStack: true
networking:
  podSubnet: 10.244.0.0/16,2001:db8:42:0::/56
  serviceSubnet: 10.96.0.0/16,2001:db8:42:1::/112
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: "10.100.0.1"
  bindPort: 6443
nodeRegistration:
  kubeletExtraArgs:
    node-ip: 10.100.0.2,fd00:1:2:3::2

advertiseAddress in InitConfiguration specifies the IP address that the API Server will advertise it is listening on. The value of advertiseAddress equals the --apiserver-advertise-address flag of kubeadm init

Run kubeadm to initiate the dual-stack control plane node:

kubeadm init --config=kubeadm-config.yaml

Currently, the kube-controller-manager flags --node-cidr-mask-size-ipv4|--node-cidr-mask-size-ipv6 are being left with default values. See enable IPv4/IPv6 dual stack.

Note: The --apiserver-advertise-address flag does not support dual-stack.

Join a node to dual-stack cluster

Before joining a node, make sure that the node has IPv6 routable network interface and allows IPv6 forwarding.

Here is an example kubeadm configuration file kubeadm-config.yaml for joining a worker node to the cluster.

apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
discovery:
  bootstrapToken:
    apiServerEndpoint: 10.100.0.1:6443
nodeRegistration:
  kubeletExtraArgs:
    node-ip: 10.100.0.3,fd00:1:2:3::3

Also, here is an example kubeadm configuration file kubeadm-config.yaml for joining another control plane node to the cluster.

apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
controlPlane:
  localAPIEndpoint:
    advertiseAddress: "10.100.0.2"
    bindPort: 6443
discovery:
  bootstrapToken:
    apiServerEndpoint: 10.100.0.1:6443
nodeRegistration:
  kubeletExtraArgs:
    node-ip: 10.100.0.4,fd00:1:2:3::4

advertiseAddress in JoinConfiguration.controlPlane specifies the IP address that the API Server will advertise it is listening on. The value of advertiseAddress equals the --apiserver-advertise-address flag of kubeadm join.

kubeadm join --config=kubeadm-config.yaml ...

Create a single-stack cluster

Note: Enabling the dual-stack feature doesn't mean that you need to use dual-stack addressing.
You can deploy a single-stack cluster that has the dual-stack networking feature enabled.

In 1.21 the IPv6DualStack feature is Beta and the feature gate is defaulted to true. To disable the feature you must configure the feature gate to false. Note that once the feature is GA, the feature gate will be removed.

kubeadm init --feature-gates IPv6DualStack=false

To make things more clear, here is an example kubeadm configuration file kubeadm-config.yaml for the single-stack control plane node.

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
featureGates:
  IPv6DualStack: false
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/16

What's next

• Validate IPv4/IPv6 dual-stack networking
• Read about Dual-stack cluster networking

3.2.2 - Installing Kubernetes with kops

This quickstart shows you how to easily install a Kubernetes cluster on AWS. It uses a tool called kops.

kops is an automated provisioning system:

• Fully automated installation
• Uses DNS to identify clusters
• Self-healing: everything runs in Auto-Scaling Groups
• Multiple OS support (Debian, Ubuntu 16.04 supported, CentOS & RHEL, Amazon Linux and CoreOS) - see the images.md
• High-Availability support - see the high_availability.md
• Can directly provision, or generate terraform manifests - see the terraform.md

Before you begin

• You must have kubectl installed.

• You must install kops on a 64-bit (AMD64 and Intel 64) device architecture.

• You must have an AWS account, generate IAM keys and configure them. The IAM user will need adequate permissions.

Creating a cluster

(1/5) Install kops

Installation

Download kops from the releases page (it is also convenient to build from source):

• macOS
• Linux

curl -LO https://github.com/kubernetes/kops/releases/download/$(curl -s https://api.github.com/repos/kubernetes/kops/releases/latest | grep tag_name | cut -d '"' -f 4)/kops-darwin-amd64

$(curl -s https://api.github.com/repos/kubernetes/kops/releases/latest | grep tag_name | cut -d '"' -f 4)

curl -LO https://github.com/kubernetes/kops/releases/download/v1.20.0/kops-darwin-amd64

chmod +x kops-darwin-amd64

sudo mv kops-darwin-amd64 /usr/local/bin/kops

brew update && brew install kops

curl -LO https://github.com/kubernetes/kops/releases/download/$(curl -s https://api.github.com/repos/kubernetes/kops/releases/latest | grep tag_name | cut -d '"' -f 4)/kops-linux-amd64

$(curl -s https://api.github.com/repos/kubernetes/kops/releases/latest | grep tag_name | cut -d '"' -f 4)

curl -LO https://github.com/kubernetes/kops/releases/download/v1.20.0/kops-linux-amd64

chmod +x kops-linux-amd64

sudo mv kops-linux-amd64 /usr/local/bin/kops

brew update && brew install kops

(2/5) Create a route53 domain for your cluster

kops uses DNS for discovery, both inside the cluster and outside, so that you can reach the kubernetes API server from clients.

kops has a strong opinion on the cluster name: it should be a valid DNS name. By doing so you will no longer get your clusters confused, you can share clusters with your colleagues unambiguously, and you can reach them without relying on remembering an IP address.

You can, and probably should, use subdomains to divide your clusters. As our example we will use useast1.dev.example.com. The API server endpoint will then be api.useast1.dev.example.com.

A Route53 hosted zone can serve subdomains. Your hosted zone could be useast1.dev.example.com, but also dev.example.com or even example.com. kops works with any of these, so typically you choose for organization reasons (e.g. you are allowed to create records under dev.example.com, but not under example.com).

Let's assume you're using dev.example.com as your hosted zone. You create that hosted zone using the normal process, or with a command such as aws route53 create-hosted-zone --name dev.example.com --caller-reference 1.

You must then set up your NS records in the parent domain, so that records in the domain will resolve. Here, you would create NS records in example.com for dev. If it is a root domain name you would configure the NS records at your domain registrar (e.g. example.com would need to be configured where you bought example.com).

Verify your route53 domain setup (it is the #1 cause of problems!). You can double-check that your cluster is configured correctly if you have the dig tool by running:

dig NS dev.example.com

You should see the 4 NS records that Route53 assigned your hosted zone.

(3/5) Create an S3 bucket to store your clusters state

kops lets you manage your clusters even after installation. To do this, it must keep track of the clusters that you have created, along with their configuration, the keys they are using etc. This information is stored in an S3 bucket. S3 permissions are used to control access to the bucket.

Multiple clusters can use the same S3 bucket, and you can share an S3 bucket between your colleagues that administer the same clusters - this is much easier than passing around kubecfg files. But anyone with access to the S3 bucket will have administrative access to all your clusters, so you don't want to share it beyond the operations team.

So typically you have one S3 bucket for each ops team (and often the name will correspond to the name of the hosted zone above!)

In our example, we chose dev.example.com as our hosted zone, so let's pick clusters.dev.example.com as the S3 bucket name.

• Export AWS_PROFILE (if you need to select a profile for the AWS CLI to work)

• Create the S3 bucket using aws s3 mb s3://clusters.dev.example.com

• You can export KOPS_STATE_STORE=s3://clusters.dev.example.com and then kops will use this location by default. We suggest putting this in your bash profile or similar.

(4/5) Build your cluster configuration

Run kops create cluster to create your cluster configuration:

kops create cluster --zones=us-east-1c useast1.dev.example.com

kops will create the configuration for your cluster. Note that it only creates the configuration, it does not actually create the cloud resources - you'll do that in the next step with a kops update cluster. This give you an opportunity to review the configuration or change it.

It prints commands you can use to explore further:

• List your clusters with: kops get cluster
• Edit this cluster with: kops edit cluster useast1.dev.example.com
• Edit your node instance group: kops edit ig --name=useast1.dev.example.com nodes
• Edit your master instance group: kops edit ig --name=useast1.dev.example.com master-us-east-1c

If this is your first time using kops, do spend a few minutes to try those out! An instance group is a set of instances, which will be registered as kubernetes nodes. On AWS this is implemented via auto-scaling-groups. You can have several instance groups, for example if you wanted nodes that are a mix of spot and on-demand instances, or GPU and non-GPU instances.

(5/5) Create the cluster in AWS

Run "kops update cluster" to create your cluster in AWS:

kops update cluster useast1.dev.example.com --yes

That takes a few seconds to run, but then your cluster will likely take a few minutes to actually be ready. kops update cluster will be the tool you'll use whenever you change the configuration of your cluster; it applies the changes you have made to the configuration to your cluster - reconfiguring AWS or kubernetes as needed.

For example, after you kops edit ig nodes, then kops update cluster --yes to apply your configuration, and sometimes you will also have to kops rolling-update cluster to roll out the configuration immediately.

Without --yes, kops update cluster will show you a preview of what it is going to do. This is handy for production clusters!

Explore other add-ons

See the list of add-ons to explore other add-ons, including tools for logging, monitoring, network policy, visualization, and control of your Kubernetes cluster.

Cleanup

• To delete your cluster: kops delete cluster useast1.dev.example.com --yes

What's next

• Learn more about Kubernetes concepts and kubectl.
• Learn more about kops advanced usage for tutorials, best practices and advanced configuration options.
• Follow kops community discussions on Slack: community discussions
• Contribute to kops by addressing or raising an issue GitHub Issues

3.2.3 - Installing Kubernetes with Kubespray

This quickstart helps to install a Kubernetes cluster hosted on GCE, Azure, OpenStack, AWS, vSphere, Packet (bare metal), Oracle Cloud Infrastructure (Experimental) or Baremetal with Kubespray.

Kubespray is a composition of Ansible playbooks, inventory, provisioning tools, and domain knowledge for generic OS/Kubernetes clusters configuration management tasks. Kubespray provides:

• a highly available cluster
• composable attributes
• support for most popular Linux distributions
  • Ubuntu 16.04, 18.04, 20.04
  • CentOS/RHEL/Oracle Linux 7, 8
  • Debian Buster, Jessie, Stretch, Wheezy
  • Fedora 31, 32
  • Fedora CoreOS
  • openSUSE Leap 15
  • Flatcar Container Linux by Kinvolk
• continuous integration tests

To choose a tool which best fits your use case, read this comparison to kubeadm and kops.

Creating a cluster

(1/5) Meet the underlay requirements

Provision servers with the following requirements:

• Ansible v2.9 and python-netaddr is installed on the machine that will run Ansible commands
• Jinja 2.11 (or newer) is required to run the Ansible Playbooks
• The target servers must have access to the Internet in order to pull docker images. Otherwise, additional configuration is required (See Offline Environment)
• The target servers are configured to allow IPv4 forwarding
• Your ssh key must be copied to all the servers part of your inventory
• The firewalls are not managed, you'll need to implement your own rules the way you used to. in order to avoid any issue during deployment you should disable your firewall
• If kubespray is ran from non-root user account, correct privilege escalation method should be configured in the target servers. Then the ansible_become flag or command parameters --become or -b should be specified

Kubespray provides the following utilities to help provision your environment:

• Terraform scripts for the following cloud providers:
  • AWS
  • OpenStack
  • Packet

(2/5) Compose an inventory file

After you provision your servers, create an inventory file for Ansible. You can do this manually or via a dynamic inventory script. For more information, see "Building your own inventory".

(3/5) Plan your cluster deployment

Kubespray provides the ability to customize many aspects of the deployment:

• Choice deployment mode: kubeadm or non-kubeadm
• CNI (networking) plugins
• DNS configuration
• Choice of control plane: native/binary or containerized
• Component versions
• Calico route reflectors
• Component runtime options
  • Docker
  • containerd
  • CRI-O
• Certificate generation methods

Kubespray customizations can be made to a variable file. If you are getting started with Kubespray, consider using the Kubespray defaults to deploy your cluster and explore Kubernetes.

(4/5) Deploy a Cluster

Next, deploy your cluster:

Cluster deployment using ansible-playbook.

ansible-playbook -i your/inventory/inventory.ini cluster.yml -b -v \
  --private-key=~/.ssh/private_key

Large deployments (100+ nodes) may require specific adjustments for best results.

(5/5) Verify the deployment

Kubespray provides a way to verify inter-pod connectivity and DNS resolve with Netchecker. Netchecker ensures the netchecker-agents pods can resolve DNS requests and ping each over within the default namespace. Those pods mimic similar behavior of the rest of the workloads and serve as cluster health indicators.

Cluster operations

Kubespray provides additional playbooks to manage your cluster: scale and upgrade.

Scale your cluster

You can add worker nodes from your cluster by running the scale playbook. For more information, see "Adding nodes". You can remove worker nodes from your cluster by running the remove-node playbook. For more information, see "Remove nodes".

Upgrade your cluster

You can upgrade your cluster by running the upgrade-cluster playbook. For more information, see "Upgrades".

Cleanup

You can reset your nodes and wipe out all components installed with Kubespray via the reset playbook.

Caution: When running the reset playbook, be sure not to accidentally target your production cluster!

Feedback

• Slack Channel: #kubespray (You can get your invite here)
• GitHub Issues

What's next

Check out planned work on Kubespray's roadmap.

3.3 - Turnkey Cloud Solutions

This page provides a list of Kubernetes certified solution providers. From each provider page, you can learn how to install and setup production ready clusters.

3.4 - Windows in Kubernetes

3.4.1 - Intro to Windows support in Kubernetes

Windows applications constitute a large portion of the services and applications that run in many organizations. Windows containers provide a modern way to encapsulate processes and package dependencies, making it easier to use DevOps practices and follow cloud native patterns for Windows applications. Kubernetes has become the defacto standard container orchestrator, and the release of Kubernetes 1.14 includes production support for scheduling Windows containers on Windows nodes in a Kubernetes cluster, enabling a vast ecosystem of Windows applications to leverage the power of Kubernetes. Organizations with investments in Windows-based applications and Linux-based applications don't have to look for separate orchestrators to manage their workloads, leading to increased operational efficiencies across their deployments, regardless of operating system.

Windows containers in Kubernetes

To enable the orchestration of Windows containers in Kubernetes, include Windows nodes in your existing Linux cluster. Scheduling Windows containers in Pods on Kubernetes is similar to scheduling Linux-based containers.

In order to run Windows containers, your Kubernetes cluster must include multiple operating systems, with control plane nodes running Linux and workers running either Windows or Linux depending on your workload needs. Windows Server 2019 is the only Windows operating system supported, enabling Kubernetes Node on Windows (including kubelet, container runtime, and kube-proxy). For a detailed explanation of Windows distribution channels see the Microsoft documentation.

Note: The Kubernetes control plane, including the master components, continues to run on Linux. There are no plans to have a Windows-only Kubernetes cluster.

Note: In this document, when we talk about Windows containers we mean Windows containers with process isolation. Windows containers with Hyper-V isolation is planned for a future release.

Supported Functionality and Limitations

Supported Functionality

Windows OS Version Support

Refer to the following table for Windows operating system support in Kubernetes. A single heterogeneous Kubernetes cluster can have both Windows and Linux worker nodes. Windows containers have to be scheduled on Windows nodes and Linux containers on Linux nodes.

Note: Information on the different Windows Server servicing channels including their support models can be found at Windows Server servicing channels.

Note: We don't expect all Windows customers to update the operating system for their apps frequently. Upgrading your applications is what dictates and necessitates upgrading or introducing new nodes to the cluster. For the customers that chose to upgrade their operating system for containers running on Kubernetes, we will offer guidance and step-by-step instructions when we add support for a new operating system version. This guidance will include recommended upgrade procedures for upgrading user applications together with cluster nodes. Windows nodes adhere to Kubernetes version-skew policy (node to control plane versioning) the same way as Linux nodes do today.

Note: The Windows Server Host Operating System is subject to the Windows Server licensing. The Windows Container images are subject to the Supplemental License Terms for Windows containers.

Note: Windows containers with process isolation have strict compatibility rules, where the host OS version must match the container base image OS version. Once we support Windows containers with Hyper-V isolation in Kubernetes, the limitation and compatibility rules will change.

Pause Image

Microsoft maintains a Windows pause infrastructure container at mcr.microsoft.com/oss/kubernetes/pause:1.4.1.

Compute

From an API and kubectl perspective, Windows containers behave in much the same way as Linux-based containers. However, there are some notable differences in key functionality which are outlined in the limitation section.

Key Kubernetes elements work the same way in Windows as they do in Linux. In this section, we talk about some of the key workload enablers and how they map to Windows.

• Pods

  A Pod is the basic building block of Kubernetes–the smallest and simplest unit in the Kubernetes object model that you create or deploy. You may not deploy Windows and Linux containers in the same Pod. All containers in a Pod are scheduled onto a single Node where each Node represents a specific platform and architecture. The following Pod capabilities, properties and events are supported with Windows containers:

  • Single or multiple containers per Pod with process isolation and volume sharing
  • Pod status fields
  • Readiness and Liveness probes
  • postStart & preStop container lifecycle events
  • ConfigMap, Secrets: as environment variables or volumes
  • EmptyDir
  • Named pipe host mounts
  • Resource limits

• Controllers

  Kubernetes controllers handle the desired state of Pods. The following workload controllers are supported with Windows containers:

  • ReplicaSet
  • ReplicationController
  • Deployments
  • StatefulSets
  • DaemonSet
  • Job
  • CronJob

• Services

  A Kubernetes Service is an abstraction which defines a logical set of Pods and a policy by which to access them - sometimes called a micro-service. You can use services for cross-operating system connectivity. In Windows, services can utilize the following types, properties and capabilities:

  • Service Environment variables
  • NodePort
  • ClusterIP
  • LoadBalancer
  • ExternalName
  • Headless services

Pods, Controllers and Services are critical elements to managing Windows workloads on Kubernetes. However, on their own they are not enough to enable the proper lifecycle management of Windows workloads in a dynamic cloud native environment. We added support for the following features:

• Pod and container metrics
• Horizontal Pod Autoscaler support
• kubectl Exec
• Resource Quotas
• Scheduler preemption

Container Runtime

Docker EE

Docker EE-basic 19.03+ is the recommended container runtime for all Windows Server versions. This works with the dockershim code included in the kubelet.

CRI-ContainerD

ContainerD 1.4.0+ can also be used as the container runtime for Windows Kubernetes nodes.

Learn how to install ContainerD on a Windows.

Caution: There is a known limitation when using GMSA with ContainerD to access Windows network shares which requires a kernel patch. Updates to address this limitation are currently available for Windows Server, Version 2004 and will be available for Windows Server 2019 in early 2021. Check for updates on the Microsoft Windows Containers issue tracker.

Persistent Storage

Kubernetes volumes enable complex applications, with data persistence and Pod volume sharing requirements, to be deployed on Kubernetes. Management of persistent volumes associated with a specific storage back-end or protocol includes actions such as: provisioning/de-provisioning/resizing of volumes, attaching/detaching a volume to/from a Kubernetes node and mounting/dismounting a volume to/from individual containers in a pod that needs to persist data. The code implementing these volume management actions for a specific storage back-end or protocol is shipped in the form of a Kubernetes volume plugin. The following broad classes of Kubernetes volume plugins are supported on Windows:

In-tree Volume Plugins

Code associated with in-tree volume plugins ship as part of the core Kubernetes code base. Deployment of in-tree volume plugins do not require installation of additional scripts or deployment of separate containerized plugin components. These plugins can handle: provisioning/de-provisioning and resizing of volumes in the storage backend, attaching/detaching of volumes to/from a Kubernetes node and mounting/dismounting a volume to/from individual containers in a pod. The following in-tree plugins support Windows nodes:

• awsElasticBlockStore
• azureDisk
• azureFile
• gcePersistentDisk
• vsphereVolume

FlexVolume Plugins

Code associated with FlexVolume plugins ship as out-of-tree scripts or binaries that need to be deployed directly on the host. FlexVolume plugins handle attaching/detaching of volumes to/from a Kubernetes node and mounting/dismounting a volume to/from individual containers in a pod. Provisioning/De-provisioning of persistent volumes associated with FlexVolume plugins may be handled through an external provisioner that is typically separate from the FlexVolume plugins. The following FlexVolume plugins, deployed as powershell scripts on the host, support Windows nodes:

• SMB
• iSCSI

CSI Plugins

Code associated with CSI plugins ship as out-of-tree scripts and binaries that are typically distributed as container images and deployed using standard Kubernetes constructs like DaemonSets and StatefulSets. CSI plugins handle a wide range of volume management actions in Kubernetes: provisioning/de-provisioning/resizing of volumes, attaching/detaching of volumes to/from a Kubernetes node and mounting/dismounting a volume to/from individual containers in a pod, backup/restore of persistent data using snapshots and cloning. CSI plugins typically consist of node plugins (that run on each node as a DaemonSet) and controller plugins.

CSI node plugins (especially those associated with persistent volumes exposed as either block devices or over a shared file-system) need to perform various privileged operations like scanning of disk devices, mounting of file systems, etc. These operations differ for each host operating system. For Linux worker nodes, containerized CSI node plugins are typically deployed as privileged containers. For Windows worker nodes, privileged operations for containerized CSI node plugins is supported using csi-proxy, a community-managed, stand-alone binary that needs to be pre-installed on each Windows node. Please refer to the deployment guide of the CSI plugin you wish to deploy for further details.

Networking

Networking for Windows containers is exposed through CNI plugins. Windows containers function similarly to virtual machines in regards to networking. Each container has a virtual network adapter (vNIC) which is connected to a Hyper-V virtual switch (vSwitch). The Host Networking Service (HNS) and the Host Compute Service (HCS) work together to create containers and attach container vNICs to networks. HCS is responsible for the management of containers whereas HNS is responsible for the management of networking resources such as:

• Virtual networks (including creation of vSwitches)
• Endpoints / vNICs
• Namespaces
• Policies (Packet encapsulations, Load-balancing rules, ACLs, NAT'ing rules, etc.)

The following service spec types are supported:

• NodePort
• ClusterIP
• LoadBalancer
• ExternalName

Network modes

Windows supports five different networking drivers/modes: L2bridge, L2tunnel, Overlay, Transparent, and NAT. In a heterogeneous cluster with Windows and Linux worker nodes, you need to select a networking solution that is compatible on both Windows and Linux. The following out-of-tree plugins are supported on Windows, with recommendations on when to use each CNI:

As outlined above, the Flannel CNI meta plugin is also supported on Windows via the VXLAN network backend (alpha support ; delegates to win-overlay) and host-gateway network backend (stable support; delegates to win-bridge). This plugin supports delegating to one of the reference CNI plugins (win-overlay, win-bridge), to work in conjunction with Flannel daemon on Windows (Flanneld) for automatic node subnet lease assignment and HNS network creation. This plugin reads in its own configuration file (cni.conf), and aggregates it with the environment variables from the FlannelD generated subnet.env file. It then delegates to one of the reference CNI plugins for network plumbing, and sends the correct configuration containing the node-assigned subnet to the IPAM plugin (e.g. host-local).

For the node, pod, and service objects, the following network flows are supported for TCP/UDP traffic:

• Pod -> Pod (IP)
• Pod -> Pod (Name)
• Pod -> Service (Cluster IP)
• Pod -> Service (PQDN, but only if there are no ".")
• Pod -> Service (FQDN)
• Pod -> External (IP)
• Pod -> External (DNS)
• Node -> Pod
• Pod -> Node

IP address management (IPAM)

The following IPAM options are supported on Windows:

• Host-local
• HNS IPAM (Inbox platform IPAM, this is a fallback when no IPAM is set)
• Azure-vnet-ipam (for azure-cni only)

Load balancing and Services

On Windows, you can use the following settings to configure Services and load balancing behavior:

IPv4/IPv6 dual-stack

You can enable IPv4/IPv6 dual-stack networking for l2bridge networks using the IPv6DualStack feature gate. See enable IPv4/IPv6 dual stack for more details.

Note: On Windows, using IPv6 with Kubernetes require Windows Server, version 2004 (kernel version 10.0.19041.610) or later.