Kubernetes

Managing a lot of containers leads to the need of a management tool. Kubernetes is one of them (not the only one, but the most famous, other examples are docker swarm, hashicorps nomad, etc). It provides an API, through which the required containers are defined and then spun up on nodes in the kubernetes cluster.

Nomenclature

• image: the files of a piece of software
• container: a running image, restricted by cgroups and other magic to isolate from the node
• node: the system running containers
• pod: a group of tightly related containers
• deployment: a group of pods

client

As of writing (2020-04) the only client system is asl503. And please adjust all the "my-" to something unique to avoid conflicts with others going through the tutorial.

the client to connect to kubernetes is kubectl.

we are running multiple clusters. api.in.a.k8s.acc.gsi.de might be the right one (or this documentation might be outdated and it is another letter than a)

configuration

we need to configure which server to use and how to authenticate

# define kubernetes cluster
# if your computer does not trust the acc-ca you need to add the flag --insecure-skip-tls-verify
kubectl config set-cluster acc --server=https://api.in.a.k8s.acc.gsi.de
# define a user
kubectl config set-credentials oidc --exec-command=kubectl-oidc_login_python
kubectl config set-credentials oidc --exec-api-version=client.authentication.k8s.io/v1beta1
# define a named context referencing the just defined values
kubectl config set-context acc --cluster=acc --user=oidc
# optionaly set a default namespace to reduce typing
kubectl config set-context acc --namespace=MYNAMESPACE
# activate context
kubectl config use-context acc

All this ends up in ~/.kube/config

Kubernetes logins use openid connect. The login client kubectl-oidc_login_python will connect to https://id.acc.gsi.de/auth/realms/acc to retrieve a token. Login happens either via kerberos ticket or interactivly prompting you for a password. The token is then cached until it expires. You can run kubectl-oidc_login_python --debug to troubleshoot problems.

Note: You can't create new namespaces and you don't have permissions on all namespaces. To get a list of namespaces kubectl get namespaces.

Objects

Kubernetes manages objects. These objects are

pod

we already know how to work with local containers. Now we run containers on a kubernetes cluster.

Kubernetes does not directly manage a container. It manages a pod. A pod holds an ip address, one or more containers and (ephemeral) files shared between these containers (a volume). As of writing 2020-04 we don't support persistent volumes.

For example you might want to run a cleanup job every hour. If your application does not manage this internally, on a normal system you would add a cron job. As containers should be a single process, a second container will be used to run cron or a simple sleep loop that triggers the cleanup. The application and the cleanup container should always run together, they form a pod. At the moment i would think most of our gsi applications are a single container per pod.

we start with a busybox container, that prints a note, sleeps for 300 seconds and then terminates. Running this localy would be something like

[handel@vmlb016 ~]$ podman run -ti busybox /bin/sh -c 'echo hello world; sleep 300'
hello world
[handel@vmlb016 ~]$ 

But we want to run it on our kubernetes cluster.

create

First create a description for our first pod my-pod.yaml. Be sure to replace "my-" with something unique to not conflict with others going through the tutorial.

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  containers:
  - name: my-app-container1
    image: busybox
    command: ['/bin/sh', '-c', 'echo hello world; sleep 300']

we declare a pod, add some metadata to it (more about labels later). Then there is one container with a name, an image and the command it should execute. Note that the command is not run through a shell, so splitting it at whitespace must be done in the definition.

send the definition to the cluster

[handel@vmlb016 ~]$ kubectl apply -f my-pod.yaml 
pod/my-app created
[handel@vmlb016 ~]$ 

We set up our default namespace during the configuration. If you did not do this, you need to specify the namespace with -n my-namespace. You won't have permissions to access the namespace "default"

inspect

which pods are running?

# list all pods
[handel@vmlb016 ~]$ kubectl get pods
NAME     READY   STATUS    RESTARTS   AGE
my-app   1/1     Running   0          3m55s

# more details
[handel@vmlb016 ~]$ kubectl get pods -o wide
NAME     READY   STATUS    RESTARTS   AGE    IP            NODE      NOMINATED NODE   READINESS GATES
my-app   1/1     Running   0          4m4s   10.253.3.10   vmlb044              

# only a single pod (as we know the name of the pod)
[handel@vmlb016 ~]$ kubectl get pod my-app
NAME     READY   STATUS    RESTARTS   AGE
my-app   1/1     Running   0          4m8s

so there is a pod, it is running, it has been running for a while. If we are interessted we could get a few more details.

We can also get the full definition of a running pod

[handel@vmlb016 ~]$ kubectl get pod my-app -o yaml
apiVersion: v1
kind: Pod
metadata:
...

what is the output of the pod?

[handel@vmlb016 ~]$ kubectl logs my-app -c my-app-container1
hello world

# logtail
[handel@vmlb016 ~]$ kubectl logs -f my-app -c my-app-container1
hello world

You remember that normaly there is a single container per pod? If there is only one container, specifying it is optional

[handel@vmlb016 ~]$ kubectl logs my-app 
hello world

and all further example assume there is only one container and leave the container name out.

restart / terminate

what happens if the sleep ends? A local container would terminate and be done. Kubernetes assumes that any defined pod has to run. It will automatically restart it. If it terminates to frequently (runtime less than ten minutes), there will be exponentially increasing delays (up to five minutes) before a restart happens.

[handel@vmlb016 ~]$ kubectl get pod my-app
NAME     READY   STATUS             RESTARTS   AGE
my-app   0/1     CrashLoopBackOff   4          27m

we can get details about this crashing pod with describe

[handel@vmlb016 ~]$ kubectl describe pod my-app
...
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 16 Apr 2020 17:07:40 +0200
      Finished:     Thu, 16 Apr 2020 17:12:40 +0200
...

and the output of the crashed (previous) container

[handel@vmlb016 ~]$ kubectl logs -p my-app

interact

a container should only run one command, but sometimes it is necessary to poke around in it.

[handel@vmlb016 ~]$ kubectl exec my-app -- ls /
bin
...

or even get an interactive shell

[handel@vmlb016 ~]$ kubectl exec my-app -ti -- /bin/sh
/ # hostname
my-app
/ # ps
PID   USER     COMMAND
    1 root     sh -c echo hello world; sleep 300
    7 root     sleep 300
   27 root     /bin/sh
   35 root     ps
/ # exit
[handel@vmlb016 ~]$ 

Note: there is no need to run an ssh daemon in your containers. As you can see, you can get a shell through kubectl. In addition it is against the rule of one process per container.

update

deliberately i did not show how to create pods without a descriptor file. We can now easily update our definition and fix an oversight. We did not specify an exact version of our container image. As a result we automaticly use latest, which for productive containers might not be the desired version

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  containers:
  - name: my-app-container1
    image: busybox:1.34
    command: ['/bin/sh', '-c', 'echo hello world; sleep 300']

If we apply it, kubernetes will stop the existing container and create a new one with the specified image

[handel@vmlb016 ~]$ kubectl apply -f my-pod.yaml 
pod/my-app configured
handel@vmlb016 ~]$ kubectl describe pod my-app|grep Image:
    Image:         busybox:1.34

Note: you can't modify the command this way. But deployments can.

wipe

[handel@vmlb016 ~]$ kubectl delete pod my-app
pod "my-app" deleted

no surprise here. Execpt the delete may will take a while. The configuration change is going through a scheduler and waits for the processes to be cleaned up.

deployment

updating pods is limited. For example you can't change the command. A Deployment solves this by using a template for the pod. Kubernetes will adjust the running pods until they match the requested status.

The descriptor my-deployment.yaml. Again change my-deployment to something unique.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  labels:
    app: my-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-deployment
  template:
    metadata:
      labels:
        app: my-deployment
    spec:
      containers:
      - name: my-deployment-container1
        image: busybox:1.34
        command: ['sh', '-c', 'echo hello world; sleep 300']

Our container definition is now wrapped inside a template.

Apply it to the cluster and list what's running

[handel@vmlb016 ~]$ kubectl apply -f my-deployment.yaml 
deployment.apps/my-deployment created

[handel@vmlb016 ~]$ kubectl get deployments
NAME            READY   UP-TO-DATE   AVAILABLE   AGE
my-deployment   1/1     1            1           1m

Wait, a deployment uses containers, but the smallest manageable unit is a pod, so there should also be pods, right? Checking:

[handel@vmlb016 ~]$ kubectl get pods 
NAME                             READY   STATUS    RESTARTS   AGE
my-deployment-65fb79885c-rc5h2   1/1     Running   0          1m

Now what is this label thing? A deployment needs to find the pods belonging to it. This is done using a selector which searches for matching pods.

If you did not wipe your pods from pod tutorial, they would still be running and have a label app=my-app. And if this descriptor would use the same label, the deployment would find and adopt the already running pods instead of creating new ones.

updating

I we want to change the command in a normal pod definition. we need to stop the existing pod and create a new one. A deployment can do this on the fly.

let's change the command in the descriptor

...
       command: ['sh', '-c', 'echo hello world, i am; hostname; sleep 300']
...

and apply it and directly check the pods

[handel@vmlb016 ~]$ kubectl apply -f my-deployment.yaml 
deployment.apps/my-deployment configured

[handel@vmlb016 ~]$ kubectl get pods
NAME                             READY   STATUS              RESTARTS   AGE
my-deployment-748df4b7b4-vqdp2   1/1     Running             0          45s
my-deployment-844cb55b48-hvbrv   0/1     ContainerCreating   0          0s

Kubernetes creates a new pod, waits till it is up and running and then terminates the old pod. Depending how quick you are you can watch the containers go though their lifecycle (Creating, Running, Terminating, etc). There is always a running container

replicas

The replica field is explained as it is a required field and has to be specified in the descriptor. Currently i can think of very few containers at gsi that would benefit of this. Please be be very careful, it is easy to consume huge amounts of resources using this feature

If your container is stateless and can run in parallel we could run multiple instances. Update the descriptor and set replicas to 3

[handel@vmlb016 ~]$ kubectl apply -f my-deployment.yaml 
deployment.apps/my-deployment configured

[handel@vmlb016 ~]$ kubectl get pods
NAME                             READY   STATUS              RESTARTS   AGE
my-deployment-844cb55b48-98z2f   0/1     ContainerCreating   0          0s
my-deployment-844cb55b48-f7cs9   0/1     ContainerCreating   0          0s
my-deployment-844cb55b48-hvbrv   1/1     Running             1          5m34s

[handel@vmlb016 ~]$ kubectl get deployments
NAME            READY   UP-TO-DATE   AVAILABLE   AGE
my-deployment   1/3     3            1           37m

# wait a bit
[handel@vmlb016 ~]$ kubectl get deployments
NAME            READY   UP-TO-DATE   AVAILABLE   AGE
my-deployment   3/3     3            3           40m

Kuberenetes created two additional pods and other a short while all three are ready.

wipe

First idea, just delete the pods

[handel@vmlb016 ~]$ kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
my-deployment-844cb55b48-5pjwg   1/1     Running   0          34s
my-deployment-844cb55b48-758cn   1/1     Running   0          34s
my-deployment-844cb55b48-hmtnt   1/1     Running   0          34s
[handel@vmlb016 ~]$ kubectl delete pod my-deployment-844cb55b48-5pjwg
pod "my-deployment-844cb55b48-5pjwg" deleted
[handel@vmlb016 ~]$ kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
my-deployment-844cb55b48-758cn   1/1     Running   0          86s
my-deployment-844cb55b48-h8f5p   1/1     Running   0          45s
my-deployment-844cb55b48-hmtnt   1/1     Running   0          86s

we deleted pod 5pjwg, but kubernetes just created a new one, as the deployment wants to have three replicas. We need to delete the deployment.

[handel@vmlb016 ~]$ kubectl delete deployment my-deployment
deployment.apps "my-deployment" deleted