While building a simple install script for Tigera’s CNX product, I received equally emphatic advice to use bash as well as golang.

I decided to build the same (subset of) functionality twice: once in bash and once again in golang.

Outcome

• The bash function took less than half the time to write and it appears equally robust and not particularly hard to read.
• The golang executable is better for scale: unit tests, strongly typed variables, scoped functionality, and more make it easier to maintain and extend safely.

Wait till pod is running - golang edition

My package builds upon the kubernetes e2e utils framework. This robust framework monitors the state of a kubernetes cluster using client-go.

package k8sutils

import (
	"fmt"
	"k8s.io/api/core/v1"
	meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/util/wait"
	"k8s.io/client-go/kubernetes"
	"k8s.io/kubernetes/pkg/client/conditions"
	"time"
)

// return a condition function that indicates whether the given pod is
// currently running
func isPodRunning(c kubernetes.Interface, podName, namespace string) wait.ConditionFunc {
	return func() (bool, error) {
		fmt.Printf(".") // progress bar!

		pod, err := c.CoreV1().Pods(namespace).Get(podName, meta_v1.GetOptions{IncludeUninitialized: true})
		if err != nil {
			return false, err
		}

		switch pod.Status.Phase {
		case v1.PodRunning:
			return true, nil
		case v1.PodFailed, v1.PodSucceeded:
			return false, conditions.ErrPodCompleted
		}
		return false, nil
	}
}

// Poll up to timeout seconds for pod to enter running state.
// Returns an error if the pod never enters the running state.
func waitForPodRunning(c kubernetes.Interface, namespace, podName string, timeout time.Duration) error {
	return wait.PollImmediate(time.Second, timeout, isPodRunning(c, podName, namespace))
}

// Returns the list of currently scheduled or running pods in `namespace` with the given selector
func ListPods(c kubernetes.Interface, namespace, selector string) (*v1.PodList, error) {
	listOptions := meta_v1.ListOptions{IncludeUninitialized: true, LabelSelector: selector}
	podList, err := c.CoreV1().Pods(namespace).List(listOptions)

	if err != nil {
		return nil, err
	}
	return podList, nil
}

// Wait up to timeout seconds for all pods in 'namespace' with given 'selector' to enter running state.
// Returns an error if no pods are found or not all discovered pods enter running state.
func WaitForPodBySelectorRunning(c kubernetes.Interface, namespace, selector string, timeout int) error {
	podList, err := ListPods(c, namespace, selector)
	if err != nil {
		return err
	}
	if len(podList.Items) == 0 {
		return fmt.Errorf("no pods in %s with selector %s", namespace, selector)
	}

	for _, pod := range podList.Items {
		if err := waitForPodRunning(c, namespace, pod.Name, time.Duration(timeout)*time.Second); err != nil {
			return err
		}
	}
	return nil
}

In addition to the k8sutils package above, I built a simple main executable package:

package main

import (
	"flag"
	"fmt"
	"github.com/bcreane/k8sutils"
	log "github.com/sirupsen/logrus"
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/tools/clientcmd"
	"k8s.io/client-go/util/homedir"
	"os"
	"path/filepath"
)

func main() {
	var kubeConfig *string
	if home := homedir.HomeDir(); home != "" {
		kubeConfig = flag.String("kubeconfig", filepath.Join(home, ".kube", "config"), "(optional) absolute path to the kubeconfig file")
	} else {
		kubeConfig = flag.String("kubeconfig", "", "absolute path to the kubeconfig file")
	}
	var namespace = flag.String("namespace", "default", "namespace")
	var selector = flag.String("selector", "", "pod selector")
	var timeout = flag.Int("timeout", 30, "timeout in seconds")
	flag.Parse()

	config, err := clientcmd.BuildConfigFromFlags("", *kubeConfig)
	if err != nil {
		panic(err)
	}
	clientSet, err := kubernetes.NewForConfig(config)
	if err != nil {
		panic(err)
	}

	// Block up to timeout seconds for listed pods in namespace/selector to enter running state
	err = k8sutils.WaitForPodBySelectorRunning(clientSet, *namespace, *selector, *timeout)
	if err != nil {
		log.Errorf("\nThe pod never entered running phase\n")
		os.Exit(1)
	}
	fmt.Printf("\nAll pods in namespace=\"%s\" with selector=\"%s\" are running!\n", *namespace, *selector)
}

Invoking the program is simple:

    # Wait up to 30 seconds for pods w/ selector and namespace to enter running phase
    ./watch --selector=k8s-app=kube-dns --namespace=kube-system --timeout=30`

This works about the same as the bash script I wrote a few weeks ago.

Take away

• Bash is quicker to write and often more condensed than golang since it builds on powerful utilities such as kubectl.
• golang is more maintainable and robust, and frankly cooler.

If you need a simple script that’s less than a few hundred lines, consider bash. Otherwise golang’s higher upfront cost but much more robust extensibility and verifiability are a better choice.

The real install script has grown to about 1,000 lines of bash. Don’t forget to allow head room for feature creep - your 50 line bash script may grow into 500 lines before you know it!

Bash is rightly critized as a train wreck when it comes to writing readable, maintainable scripts. See Stop writing shell Scripts! for a sample.

If you’re careful, it’s possible to write a bash script that is sufficiently well-written to (potentially) justify the extra maintenance costs.

I’ll solve a useful problem in both bash as well as golang, comparing up-front the costs.

First edition: bash script that blocks until a kubernetes pod is “ready,” or a timeout fires.

Some bash best practices

We’ll use a few features of “modern” bash (if that’s not too much of an oyxmoron):

• Subshells: rather than using backticks, isolating a command in a subshell is good security practice:

    status=$(ls)  # grab the output from a subshell
  

• Arithmetic:

    $((count--))  # looks like C if you squint hard enough
  

• String regexes: bash supports some python-esque operators which are concise and convenient, though you must use the clumsy [[ ]] test operator:

      if [[ $(echo "hello, fool") =~ "fool" ]]; then  # search for "fool" substring
        echo You are indeed a fool.
      fi
  

• Use local to reduce the scope of variables, e.g.:

  function fun() {
    local rabbit="marty"  # $rabbit is not visible outside the function "fun()"
    echo "${rabbit}"
  }
  
  fun             # invoke function "fun()", prints "marty"
  echo "$rabbit"  # in the global scope, "$rabbit" is uninitialized, prints ""
  

Bash is glue

Obviously bash is just the glue that binds a myriad of tools together - similar to the range of packages available to a golang program.

In this case, we’ll use the following standalone programs:

• kubectl: reports the pod’s state, optionally in json format
• jq: parses and filters kubectl’s json output

Bash script to block until a pod is ready (or timeout)

#!/bin/bash
# Brendan Creane
# Block until a pod specified by a selector is "ready"
# or a timeout occurs.

#
# podStatus() - takes a selector as argument, e.g. "k8s-app=kube-dns"
# and returns the pod "ready" status as a bool string ("true"). Note that if
# the pod is in the "pending" state, there is no containerStatus yet, so
# podStatus() returns an empty string. Success means seeing the "true"
# substring, but failure can be "false" or an empty string.
#
function podStatus() {
  local label="$1"
  local status=$(kubectl get pods --selector="${label}" -o json --all-namespaces | jq -r '.items[] | .status.containerStatuses[]? | [.name, .image, .ready|tostring] |join(":")')
  echo "${status}"
}

#
# blockUntilPodIsReady() - takes a pod selector and a timeout in seconds
# as arguements. If the pod never stabilizes, bail. Otherwise return as
# soon as the pod is "ready."
#
function blockUntilPodIsReady() {
  local label="$1"
  local secs="$2"
  local friendlyPodName="$3"

  echo -n "Waiting for \"${friendlyPodName}\" to be ready: "
  until [[ $(podStatus "${label}") =~ "true" ]]; do
    if [ "$secs" -eq 0 ]; then
      echo "\"${friendlyPodName}\" never stabilized."
      exit 1
    fi

    : $((secs--))
    echo -n .
    sleep 1
  done
}

blockUntilPodIsReady "k8s-app=kube-dns" 120 "kube-dns"  # Block until kube-dns is running & ready

Cost analysis

Depending on your familiarity with jq, kubectl and bash, writing this script could take between half an hour up to several hours. It doesn’t look much harder to read than modern typed languages such as golang.

Some of the problems with using bash include:

• Namespaces: bash code usually lives in a single file, and there’s obviously no object-oriented encapsulation or scope. Using local helps a bit since bash variables are global by default.

• Unit and functional tests: writing tests seems possible in theory, but I admit I’ve never enjoyed writing bash tests … which brings us to the final point.

• Writing bash is not fun. I’ve never experienced that feeling of easy and flow with bash that comes from writing a beautiful C++ or golang program

Next week, the golang equivalent using client-go.

Google App Engine is very easy to provision - automatic tls certs, friendly domain name, caching, scaling and robust debugging are among the useful features.

However using GAE to serve a static Jekyll site (admittedly not it’s most important use-case) has issues ranging from annoying to serious.

Before deploying a static site on GAE, make sure the issues outlined below are not blockers for you.

Deploying your Jekyll site on GAE

Jekyll creates a _site/ directory with all of your site’s markdown rendered to .html. Google App Engine takes a app.yaml description of your site and publishes the relevant files.

GAE issue: differentiating files and directories

One nice feature of Jekyll is Permalink style which let’s you link to your pages without annoying suffixes such as .html. Unfortunately GAE doesn’t handle this well.

Most webservers can handle all three of the following scenarios:

There is a workaround … you have to tell GAE about every directory and file in your site!

#!/bin/bash

# Script to generate the site yaml for GAE to serve your site
# Get all the unique file suffixes in the _site; e.g. ".txt", ".png".
# Output in form "js|html|yml", etc.
suffixes=`find _site -type f -iname \*.* -print | sed 's/.*\.//' | sort | uniq | paste -sd "|" -`

# Create a static file handler based on all the suffixes
printf -- "- url: /(.*\\.(%s))$\n" $suffixes
printf "  static_files: _site/\\\1\n"
printf "  upload: _site/(.*\\.(%s))$\n" $suffixes

Now let’s handle the case where someone forgets to append a / to a directory URL, for example: https://_site/fun when they mean to say https://_site/fun/

#!/bin/bash

# Get all the directories in the _site; e.g. "fun"
# Output in form "/dir1|/dir2", etc.
directories=`find _site -type d -print | sed 's/_site\///g' | sort | uniq | grep -v _site | paste -sd "|" -`

# Create a handler for URLs with a directory that do NOT have a terminal /.
printf "\n"
printf "# Handle any directory URLs that are missing a terminal /\n"
printf "#\n"
printf -- "- url: /(%s)$\n" $directories
printf "  static_files: _site/\\\1/index.html\n"
printf "  upload: _site/(%s)/index.html\n" $directories

Okay, this is awkward and doesn’t scale, but fortunately we have a pretty small site with fewer than 100 directories and 500 files. The generated app.yaml is big, but not impossibly big.

GAE issue: mime type for scripts or yaml?

Most webservers give you a robust pre-populated list of mime types. There are only about a dozen file suffixes in our site, e.g. yaml, txt, html, jpg. Nothing exotic.

Turns out GAE’s webserver doesn’t know what to do with .bash, .sh, or .yaml. Again, not a big deal, but our script is getting bigger:

# Serve up static files based on suffix. See:
#   https://www.iana.org/assignments/media-types/media-types.xhtml
#
# Specify mime-type for yaml files since GAE doesn't handle this correctly.
#
- url: /(.*\.(yaml|yml))$
  static_files: _site/\1
  mime_type: text/x-yaml
  upload: _site/(.*\.(yaml|yml))$

# Specify mime-type for sh files since GAE doesn't handle this correctly.
#
- url: /(.*\.(sh|bash))$
  static_files: _site/\1
  mime_type: text/x-shellscript
  upload: _site/(.*\.(sh|bash))$

# For all remaining files, let GAE infer mime-type

GAE issue: GAE sends compressed files, unexpectedly

This one drove me crazy for a little while. Trying to download a yaml file: curl -O https://my.site.io/my_kubernetes_manifest.yaml returned binary junk in the file!

It turns out GAE sometimes sends back gzipped data, even if the request header of the client doesn’t say it can accept it. But only on https, not http. And only sometimes.

There’s a one-year old bug Google frontend serves gzipped content even if the client doesn’t ask for it which captures this issue perfectly.

There are other interesting, and unresolved bugs around this area too. Try the handy search GAE issues.

The workaround is to force curl to expect a gzipped response: curl --compressed -O https://my.site.io/my_kubernetes_manifest.yaml.

Conclusion

GAE is a subpar approach for serving static content for the reasons outlined above (and others). More worryingly, the slow response to outstanding bugs implies Google’s attention is elsewhere.

Justin Krause did a nice job of reviewing GAE for python dynamic sites. GAE covered their needs nicely. As long as you’re using GAE for exactly what it was designed for, and you don’t mind working around bugs, GAE may work for you.

Serving a Jekyll site with Google App Engine

Here’s the bash script for generating the GAE site yaml in it’s entirety.

#!/bin/bash

# Script that inspects the jekyll-generated _site and emits
# a Google App Engine service specification file on stdout.

if [ ! -d _site ]; then
  echo _site/ doesn\'t exist yet, make sure you run \"jekyll build\".
  exit 1
fi

cat <<EOF
# A Google Application Engine "service" definition. The reference
# to python is a red-herring - we're just statically serving the
# contents of the _site subdirectory. GAE isn't setup to do this
# easily - we need to tell it how to handle each type of file, as
# well as infer "index.html" when we just refer to a site directory.
#
runtime: python27
api_version: 1
threadsafe: yes

handlers:
# Serve up static files based on suffix. See:
#   https://www.iana.org/assignments/media-types/media-types.xhtml
#
# Specify mime-type for yaml files since GAE doesn't handle this correctly.
#
- url: /(.*\.(yaml|yml))$
  static_files: _site/\1
  mime_type: text/x-yaml
  upload: _site/(.*\.(yaml|yml))$

# Specify mime-type for sh files since GAE doesn't handle this correctly.
#
- url: /(.*\.(sh|bash))$
  static_files: _site/\1
  mime_type: text/x-shellscript
  upload: _site/(.*\.(sh|bash))$

# For all remaining files, let GAE infer mime-type
#
EOF

# Get all the unique file suffixes in the _site; e.g. ".txt", ".png".
# Output in form "js|html|yml", etc.
suffixes=`find _site -type f -iname \*.* -print | sed 's/.*\.//' | sort | uniq | paste -sd "|" -`

# Create a static file handler based on all the suffixes
printf -- "- url: /(.*\\.(%s))$\n" $suffixes
printf "  static_files: _site/\\\1\n"
printf "  upload: _site/(.*\\.(%s))$\n" $suffixes

# Enumerate all the directories in the _site; e.g. "fun/". Output in form "/dir1|/dir2", etc.
directories=`find _site -type d -print | sed 's/_site\///g' | sort | uniq | grep -v _site | paste -sd "|" -`

# Create a handler for URLs with a directory that do NOT have a
# terminal /. This is a fail-safe in case someone misconstructs
# the URL without a terminal /.
printf "\n"
printf "# Handle any directory URLs that are missing a terminal /\n"
printf "#\n"
printf -- "- url: /(%s)$\n" $directories
printf "  static_files: _site/\\\1/index.html\n"
printf "  upload: _site/(%s)/index.html\n" $directories

cat <<EOF

# Default directory/html append rules
#
- url: /
  static_files: _site/index.html
  upload: _site/index.html

# For directories indicated by a terminal /, append "/index.html"
- url: /(.+)/
  static_files: _site/\1/index.html
  upload: _site/(.+)/index.html
  expiration: "15m"

# For md files, append ".html"
- url: /(.+[a-z0-9])
  static_files: _site/\1.html
  upload: _site/(.+[a-z]).html
  expiration: "15m"

- url: /(.+)
  static_files: _site/\1/index.html
  upload: _site/(.+)/index.html
  expiration: "15m"

- url: /(.*)
  static_files: _site/\1
  upload: _site/(.*)

libraries:
- name: webapp2
  version: "2.5.2"
EOF