commit	6aa564dac191cc832568230e4c769555f2efdb84	[log] [tgz]
author	Avi Kondareddy <avikr@google.com>	Thu Nov 15 17:49:08 2018 -0800
committer	Avi Kondareddy <avikr@google.com>	Thu Nov 15 17:50:09 2018 -0800
tree	16a58e40f75d472efe75c21cb69095614a44c318
parent	4399ee0205caa3b6d22ab6fef5b7518204453780 [diff]

tree: 16a58e40f75d472efe75c21cb69095614a44c318

README.md

Prow Presubmit for KUnit

KUnit uses Prow for presubmit and CI.

This repository contains all the code and instructions needed for generating a docker container for building and running KUnit tests on Prow, and then using that container with a Prow cluster.

The official repo for Prow is test-infra, Kubernetes' test infrastructure. It enables us to run presubmit jobs on changes to our gerrit repo against an arbitrary docker container image. We have pushed the docker image generated from here to gcr.io/kunit-presubmit/kunit. The image is pulled and deployed by our prow cluster, source is pulled into working directory of container from kunit source, and entrypoint script (kunit.sh) is ran. The script copies kunitconfig to working directory, runs kunit.py, and sends output to job artifacts that can be viewed later. Prow will comment on the Gerrit change with the status of the job (successful, a test failed, a test crashed, etc) and link to Prow URL for more details. The container is currently configured to require the inclusion of kunitconfig in repo. A convenience of this model is that updating the job image can be done without change to ProwJob configuration.

ProwJob Docker image

The docker image is configured in the Dockerfile. It is based off of Debian, with kernel build tools installed and the script ran by the prowjob included. A couple peculiarities of building/running UML in a docker container are solved in the deployment configuration discussed below under “Prow Job Specification”.

TODO(avikr@google.com): generalize for users using docker registry

To build and push, you must have Docker and Google Cloud SDK installed locally. Make sure to enable sudoless docker (fixes gcr push authentication problems). You will need to enable gcloud authentication first to push with gcloud auth configure-docker.

TODO(avikr@google.com): replace with bazel build

Now to build and push image :

# build image
docker build . gcr.io/kunit-presubmit/kunit

# confirm image is built
docker images

# push to gcr
docker push gcr.io/kunit-presubmit/kunit

To test the container locally:

# have tmp directory with source checked out to $TMP/linux
cp Dockerfile.test $TMP/Dockerfile
cd $TMP

# build test container which includes source
docker build . -t test

# run with args to handle issues with UML in Docker.
docker run --privileged --tmpfs /dev/shm:exec test

# extract log
RUN=$(docker container ls --last 1 -q)
docker cp $RUN:/artifacts/kunit.log .
cat kunit.log

# cleanup container
docker rm $RUN

Note: Testing with an interactive shell results in unexpected behaviour. Running the UML Kernel in an entrypoint script works as intended but fails in an interactive shell.

Prow Job Specification

Prow Jobs are detailed at testinfra/prow/jobs.md. The prow job specification is held in the config.yaml. It specifies the gerrit repo for prow to poll and the specifications for the container. We found that running the container privileged and mounting a emptydir at /dev/shm fixed KUnit build errors. There may be more secure methods, but as prow doesn't expose job containers to external resources, this solution results in the cleanest Dockerfile. We are using the decorated prow job which is recommended for all new prowjobs and detailed at test-infra/prow/pod-utilities.md.

Job Script

The job script and kunitconfig stored inside in the docker image are kunit.sh and kunitconfig.

Prow sets the environment variable ARTIFACTS to specify a directory that will be exported to gcloud on job completion. It additionally sends all data send to stdout/stderr to the reporting interface. Exit code of 0 signals success and 1 signals failure akin to regular shell scripts.

Prow Cluster Deployment

TODO(avikr@google.com): add cluster.yaml and explain