README: Update for building/pushing Docker image and configuring ProwJob
Change-Id: Ic3717de09641e5ceb8a2879b843967a124d5f710
Google-Bug-Id: 119262489
Signed-off-by: Avi Kondareddy <avikr@google.com>
diff --git a/README.md b/README.md
index 0ef511e..3c1af5c 100644
--- a/README.md
+++ b/README.md
@@ -2,9 +2,110 @@
KUnit uses Prow for presubmit and CI.
-TODO(avikr@google.com): add appropriate Prow links and descriptions.
-
This repository contains all the code and instructions needed for generating a
docker container for building and running KUnit tests on Prow, and then using
that container with a Prow cluster.
+The official repo for Prow is
+[test-infra](https://github.com/kubernetes/test-infra), Kubernetes' test
+infrastructure. It enables us to run presubmit jobs on changes to our gerrit
+repo against an arbitrary docker container image. We have pushed the docker
+image generated from here to gcr.io/kunit-presubmit/kunit. The image is pulled
+and deployed by our prow cluster, source is pulled into working directory of
+container from
+[kunit source](https://kunit.googlesource.com/linux/kunit/alpha/master), and
+entrypoint script (kunit.sh) is ran. The script copies kunitconfig to working
+directory, runs kunit.py, and sends output to job artifacts that can be viewed
+later. Prow will comment on the Gerrit change with the status of the job
+(successful, a test failed, a test crashed, etc) and link to Prow URL for more
+details. The container is currently configured to require the inclusion of
+kunitconfig in repo. A convenience of this model is that updating the job image
+can be done without change to ProwJob configuration.
+
+## ProwJob Docker image
+
+The docker image is configured in the
+[Dockerfile](Dockerfile). It is
+based off of Debian, with kernel build tools installed and the script ran by the
+prowjob included. A couple peculiarities of building/running UML in a docker
+container are solved in the deployment configuration discussed below under "Prow
+Job Specification".
+
+TODO(avikr@google.com): generalize for users using docker registry
+
+To build and push, you must have [Docker](
+https://docs.docker.com/install/linux/docker-ce/debian/) and [Google Cloud SDK](
+https://cloud.google.com/sdk/install) installed locally. Make sure to enable
+[sudoless docker](https://docs.docker.com/install/linux/linux-postinstall/)
+(fixes gcr push
+authentication problems). You will need to enable gcloud authentication first to
+push with `gcloud auth configure-docker`.
+
+TODO(avikr@google.com): replace with bazel build
+
+Now to build and push image :
+
+```shell
+# build image
+docker build . gcr.io/kunit-presubmit/kunit
+
+# confirm image is built
+docker images
+
+# push to gcr
+docker push gcr.io/kunit-presubmit/kunit
+```
+
+To test the container locally:
+
+```shell
+# have tmp directory with source checked out to $TMP/linux
+cp Dockerfile.test $TMP/Dockerfile
+cd $TMP
+
+# build test container which includes source
+docker build . -t test
+
+# run with args to handle issues with UML in Docker.
+docker run --privileged --tmpfs /dev/shm:exec test
+
+# extract log
+RUN=$(docker container ls --last 1 -q)
+docker cp $RUN:/artifacts/kunit.log .
+cat kunit.log
+
+# cleanup container
+docker rm $RUN
+```
+
+Note: Testing with an interactive shell results in unexpected behaviour. Running
+the UML Kernel in an entrypoint script works as intended but fails in an
+interactive shell.
+
+## Prow Job Specification
+
+Prow Jobs are detailed at
+[testinfra/prow/jobs.md](https://github.com/kubernetes/test-infra/blob/master/prow/jobs.md).
+The prow job specification is held in the
+[config.yaml](config.yaml). It
+specifies the gerrit repo for prow to poll and the specifications for the
+container. We found that running the container privileged and mounting a
+emptydir at /dev/shm fixed KUnit build errors. There may be more secure methods,
+but as prow doesn't expose job containers to external resources, this solution
+results in the cleanest Dockerfile. We are using the decorated prow job which is
+recommended for all new prowjobs and detailed at
+[test-infra/prow/pod-utilities.md](https://github.com/kubernetes/test-infra/blob/master/prow/pod-utilities.md).
+
+## Job Script
+
+The job script and kunitconfig stored inside in the docker image are
+[kunit.sh](kunit.sh) and
+[kunitconfig](kunitconfig).
+
+Prow sets the environment variable ARTIFACTS to specify a directory that will be
+exported to gcloud on job completion. It additionally sends all data send to
+stdout/stderr to the reporting interface. Exit code of 0 signals success and 1
+signals failure akin to regular shell scripts.
+
+## Prow Cluster Deployment
+TODO(avikr@google.com): add cluster.yaml and explain
