Learn Kubernetes Basics

Concepts

Edit This Page

Cloud Providers

This page explains how to manage Kubernetes running on a specific cloud provider.

kubeadm

kubeadm is a popular option for creating kubernetes clusters. kubeadm has configuration options to specify configuration information for cloud providers. For example a typical in-tree cloud provider can be configured using kubeadm as shown below:

apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
nodeRegistration:
  kubeletExtraArgs:
    cloud-provider: "openstack"
    cloud-config: "/etc/kubernetes/cloud.conf"
---
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: v1.13.0
apiServer:
  extraArgs:
    cloud-provider: "openstack"
    cloud-config: "/etc/kubernetes/cloud.conf"
  extraVolumes:
  - name: cloud
    hostPath: "/etc/kubernetes/cloud.conf"
    mountPath: "/etc/kubernetes/cloud.conf"
controllerManager:
  extraArgs:
    cloud-provider: "openstack"
    cloud-config: "/etc/kubernetes/cloud.conf"
  extraVolumes:
  - name: cloud
    hostPath: "/etc/kubernetes/cloud.conf"
    mountPath: "/etc/kubernetes/cloud.conf"

The in-tree cloud providers typically need both --cloud-provider and --cloud-config specified in the command lines for the kube-apiserver, kube-controller-manager and the kubelet. The contents of the file specified in --cloud-config for each provider is documented below as well.

For all external cloud providers, please follow the instructions on the individual repositories.

AWS

This section describes all the possible configurations which can be used when running Kubernetes on Amazon Web Services.

Node Name

The AWS cloud provider uses the private DNS name of the AWS instance as the name of the Kubernetes Node object.

Load Balancers

You can setup external load balancers to use specific features in AWS by configuring the annotations as shown below.

apiVersion: v1
kind: Service
metadata:
  name: example
  namespace: kube-system
  labels:
    run: example
  annotations:
     service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xx-xxxx-x:xxxxxxxxx:xxxxxxx/xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx #replace this value
     service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
spec:
  type: LoadBalancer
  ports:
  - port: 443
    targetPort: 5556
    protocol: TCP
  selector:
    app: example

Different settings can be applied to a load balancer service in AWS using annotations. The following describes the annotations supported on AWS ELBs:

The information for the annotations for AWS is taken from the comments on aws.go

Azure

Node Name

The Azure cloud provider uses the hostname of the node (as determined by the kubelet or overridden with --hostname-override) as the name of the Kubernetes Node object. Note that the Kubernetes Node name must match the Azure VM name.

CloudStack

Node Name

The CloudStack cloud provider uses the hostname of the node (as determined by the kubelet or overridden with --hostname-override) as the name of the Kubernetes Node object. Note that the Kubernetes Node name must match the CloudStack VM name.

GCE

Node Name

The GCE cloud provider uses the hostname of the node (as determined by the kubelet or overridden with --hostname-override) as the name of the Kubernetes Node object. Note that the first segment of the Kubernetes Node name must match the GCE instance name (e.g. a Node named kubernetes-node-2.c.my-proj.internal must correspond to an instance named kubernetes-node-2).

OpenStack

This section describes all the possible configurations which can be used when using OpenStack with Kubernetes.

Node Name

The OpenStack cloud provider uses the instance name (as determined from OpenStack metadata) as the name of the Kubernetes Node object. Note that the instance name must be a valid Kubernetes Node name in order for the kubelet to successfully register its Node object.

Services

The OpenStack cloud provider implementation for Kubernetes supports the use of these OpenStack services from the underlying cloud, where available:

Service API Version(s) Required
Block Storage (Cinder) V1†, V2, V3 No
Compute (Nova) V2 No
Identity (Keystone) V2‡, V3 Yes
Load Balancing (Neutron) V1§, V2 No
Load Balancing (Octavia) V2 No

† Block Storage V1 API support is deprecated, Block Storage V3 API support was added in Kubernetes 1.9.

‡ Identity V2 API support is deprecated and will be removed from the provider in a future release. As of the “Queens” release, OpenStack will no longer expose the Identity V2 API.

§ Load Balancing V1 API support was removed in Kubernetes 1.9.

Service discovery is achieved by listing the service catalog managed by OpenStack Identity (Keystone) using the auth-url provided in the provider configuration. The provider will gracefully degrade in functionality when OpenStack services other than Keystone are not available and simply disclaim support for impacted features. Certain features are also enabled or disabled based on the list of extensions published by Neutron in the underlying cloud.

cloud.conf

Kubernetes knows how to interact with OpenStack via the file cloud.conf. It is the file that will provide Kubernetes with credentials and location for the OpenStack auth endpoint. You can create a cloud.conf file by specifying the following details in it

Typical configuration

This is an example of a typical configuration that touches the values that most often need to be set. It points the provider at the OpenStack cloud’s Keystone endpoint, provides details for how to authenticate with it, and configures the load balancer:

[Global]
username=user
password=pass
auth-url=https://<keystone_ip>/identity/v3
tenant-id=c869168a828847f39f7f06edd7305637
domain-id=2a73b8f597c04551a0fdc8e95544be8a

[LoadBalancer]
subnet-id=6937f8fa-858d-4bc9-a3a5-18d2c957166a
Global

These configuration options for the OpenStack provider pertain to its global configuration and should appear in the [Global] section of the cloud.conf file:

When using Keystone V3 - which changes tenant to project - the tenant-id value is automatically mapped to the project construct in the API.

Load Balancer

These configuration options for the OpenStack provider pertain to the load balancer and should appear in the [LoadBalancer] section of the cloud.conf file:

Block Storage

These configuration options for the OpenStack provider pertain to block storage and should appear in the [BlockStorage] section of the cloud.conf file:

If deploying Kubernetes versions <= 1.8 on an OpenStack deployment that uses paths rather than ports to differentiate between endpoints it may be necessary to explicitly set the bs-version parameter. A path based endpoint is of the form http://foo.bar/volume while a port based endpoint is of the form http://foo.bar:xxx.

In environments that use path based endpoints and Kubernetes is using the older auto-detection logic a BS API version autodetection failed. error will be returned on attempting volume detachment. To workaround this issue it is possible to force the use of Cinder API version 2 by adding this to the cloud provider configuration:

[BlockStorage]
bs-version=v2
Metadata

These configuration options for the OpenStack provider pertain to metadata and should appear in the [Metadata] section of the cloud.conf file:

Influencing this behavior may be desirable as the metadata on the configuration drive may grow stale over time, whereas the metadata service always provides the most up to date view. Not all OpenStack clouds provide both configuration drive and metadata service though and only one or the other may be available which is why the default is to check both.

Router

These configuration options for the OpenStack provider pertain to the kubenet Kubernetes network plugin and should appear in the [Router] section of the cloud.conf file:

OVirt

Node Name

The OVirt cloud provider uses the hostname of the node (as determined by the kubelet or overridden with --hostname-override) as the name of the Kubernetes Node object. Note that the Kubernetes Node name must match the VM FQDN (reported by OVirt under <vm><guest_info><fqdn>...</fqdn></guest_info></vm>)

Photon

Node Name

The Photon cloud provider uses the hostname of the node (as determined by the kubelet or overridden with --hostname-override) as the name of the Kubernetes Node object. Note that the Kubernetes Node name must match the Photon VM name (or if overrideIP is set to true in the --cloud-config, the Kubernetes Node name must match the Photon VM IP address).

VSphere

Node Name

The VSphere cloud provider uses the detected hostname of the node (as determined by the kubelet) as the name of the Kubernetes Node object.

The --hostname-override parameter is ignored by the VSphere cloud provider.

IBM Cloud Kubernetes Service

Compute nodes

By using the IBM Cloud Kubernetes Service provider, you can create clusters with a mixture of virtual and physical (bare metal) nodes in a single zone or across multiple zones in a region. For more information, see Planning your cluster and worker node setup.

The name of the Kubernetes Node object is the private IP address of the IBM Cloud Kubernetes Service worker node instance.

Networking

The IBM Cloud Kubernetes Service provider provides VLANs for quality network performance and network isolation for nodes. You can set up custom firewalls and Calico network policies to add an extra layer of security for your cluster, or connect your cluster to your on-prem data center via VPN. For more information, see Planning in-cluster and private networking.

To expose apps to the public or within the cluster, you can leverage NodePort, LoadBalancer, or Ingress services. You can also customize the Ingress application load balancer with annotations. For more information, see Planning to expose your apps with external networking.

Storage

The IBM Cloud Kubernetes Service provider leverages Kubernetes-native persistent volumes to enable users to mount file, block, and cloud object storage to their apps. You can also use database-as-a-service and third-party add-ons for persistent storage of your data. For more information, see Planning highly available persistent storage.

Baidu Cloud Container Engine

Node Name

The Baidu cloud provider uses the private IP address of the node (as determined by the kubelet or overridden with --hostname-override) as the name of the Kubernetes Node object. Note that the Kubernetes Node name must match the Baidu VM private IP.

Feedback