Your 2nd day with Kubernetes on AWS

Your 2nd day with Kubernetes on AWS

Your 2nd day with Kubernetes on AWS

Okay, so you have a cluster up and running on AWS. Now what? Seriously, managing a Kubernetes cluster is hard. Especially if you are even thinking about keeping up with the pace of the community. The good news, is that kops makes this easy. Here a few commonly used stories on how to manage a cluster after everything is already up and running. If there is something you don’t see, that you would like, please let me know!

This tutorial assumes you were able to successfully get a cluster up and running in AWS, and you are now ready to see what else it can do.

In this tutorial we are covering 2nd day concerns for managing a Kubernetes cluster on AWS. The idea of this tutorial is to exercise some useful bits of kops functionality that you won’t see during a cluster deployment. Here we really open up kops to see what she can do (yes, kops is a girl)

In this tutorial we will be running kops 1.5.1, which can be downloaded here.

We will also be making the assumption that you have an environment setup similar to the following.

export KOPS_STATE_STORE=s3://nivenly-com-state-store

Upgrading Kubernetes with kops

Suppose you are running an older version of Kubernetes, and want to run the latest and greatest..

Here we will start off with a Kubernetes v1.4.8 cluster. We are picking an older cluster here to demonstrate the workflow in which you could upgrade your Kubernetes cluster. The project evolves quickly, and you want to be able to iterate on your clusters just as quickly. To deploy a Kubernetes v1.4.8 cluster:

kops create cluster --zones us-west-2a --kubernetes-version 1.4.8 $KOPS_NAME --yes

As the cluster is deploying, notice how kops will conveniently remind us that the version of Kubernetes that we are deploying is outdated. This is by design. We really want users to know when they are running old code.

A new kubernetes version is available: 1.5.2
Upgrading is recommended (try kops upgrade cluster)

So now we have an older version of Kubernetes running. We know this by running the following command and looking for Server Version: version.Info

kubectl version

Now, we can use the following command to see what kops suggests we should do:

kops upgrade cluster $KOPS_NAME

We can safely append --yes to the end of our command to apply the upgrade to our configuration. But what is really happening here?

When we run the upgrade command as in

kops upgrade cluster $KOPS_NAME --yes

all we are really doing is appending some values to the cluster spec. (Remember, this is the state store that is stored in S3 in YAML). Which of course can always be accessed and edited using:

kops edit cluster $KOPS_NAME

In this case you will notice how the kops upgrade cluster command conveniently changed the following line in the configuration file for us.

  kubernetesVersion: 1.5.2

We can now run a kops update cluster command as we always would, to apply the change.

kops update cluster $KOPS_NAME --yes

We can now safely roll each of our nodes to finish the upgrade. Let’s use kops rolling-update cluster to re deploy each of our nodes. This is necessary to finish the upgrade. A kops rolling update will cycle each of the instance in the autoscale group with the new configuration.

kops rolling-update cluster $KOPS_NAME --yes

We can now check the version of Kubernetes, and validate that we are in fact using the latest version.

 kubectl version 

Note: If a specific version of Kubernetes is desired, you can always use the --channel flag and specify a valid channel. An example channel can be found here.

Scaling your cluster

Suppose you would like to scale your cluster to process more work..

In this example we will start off with a very basic cluster, and turn the node count up using kops instance groups.

kops create cluster --zones us-west-2a --node-count 3 $KOPS_NAME --yes

After the cluster is deployed we can validate that we are using 3 nodes by running

kubectl get nodes

Say we want to scale our nodes from 3 to 30. We can easily do that with kops by editing the nodes instance group using:

kops edit instancegroup nodes

We can then bump our node counts up to 30

  machineType: t2.medium
  maxSize: 30
  minSize: 30
  role: Node
  - us-west-2a

We then of course need to update our newly edited configuration

kops update cluster $KOPS_NAME --yes


Kops will update the AWS ASG automatically, and poof we have a 30 node cluster.

I do actually try all of this before recommending it to anyone. So yes, I was able to actually deploy a 30 node cluster in AWS with kops.

The cluster was deployed successfully, and the primary component of lag was waiting on Amazon to deploy the instances after detecting a change in the Autoscaling group.


A quick delete command from kops, and all is well.

 kops delete cluster $KOPS_NAME --yes 

Audit your clusters

Suppose you need to know what is going on in the cloud.. and audit your infrastructure..

By design kops will never store information about the cloud resources, and will always look them up at runtime. So gaining a glimpse into what you have running currently can be a bit of a concern. There are 2 kops commands that are very useful for auditing your environment, and also auditing a single cluster.

In order to see what clusters we have running in a state store we first use the following command:

kops get clusters

Notice how we no longer have to use `$KOPS_NAME`. This is because we already have a cluster deployed, and thus should already have a working `~/.kube/config` file in place. We can infer a lot of information from the file. Now that we have a cluster name (or more!) in mind, we can use the following command:

kops toolbox dump

Which will output all the wonderful information we could want about a cluster in a format that is easy to query. It is important to note that the resources defined here are discovered using the same cluster lookup methods `kops` uses for all other cluster commands. This is a raw, and unique output of your cluster at runtime!


Thank you for reading my article. As always, I appreciate any feedback from users. So let me know how we could be better. Also feel free to check me out on GitHub for more kops updates.

Follow @kris-nova


  1. Martijn
    June 4, 2017 22:40:pm Reply

    Thanks for your excellent tutorials.

    I’d like to know how to update a 1-master 1-AZ cluster into a HA 2-AZ cluster. Is it even possible?