README.md - prow-presubmit - Git at Google

 # Prow Presubmit for KUnit

 KUnit uses Prow for presubmit and CI.

 This repository contains all the code and instructions needed for generating a
 docker container for building and running KUnit tests on Prow, and then using
 that container with a Prow cluster.

 The official repo for Prow is
 [test-infra](https://github.com/kubernetes/test-infra), Kubernetes' test
 infrastructure. It enables us to run presubmit jobs on changes to our gerrit
 repo against an arbitrary docker container image. We have pushed the docker
 image generated from here to gcr.io/kunit-presubmit/kunit. The image is pulled
 and deployed by our prow cluster, source is pulled into working directory of
 container from
 [kunit source](https://kunit.googlesource.com/linux/kunit/alpha/master), and
 entrypoint script (kunit.sh) is ran. The script copies kunitconfig to working
 directory, runs kunit.py, and sends output to job artifacts that can be viewed
 later. Prow will comment on the Gerrit change with the status of the job
 (successful, a test failed, a test crashed, etc) and link to Prow URL for more
 details. The container is currently configured to require the inclusion of
 kunitconfig in repo. A convenience of this model is that updating the job image
 can be done without change to ProwJob configuration.

 ## ProwJob Docker image

 The docker image is configured in the
 [Dockerfile](Dockerfile). It is
 based off of Debian, with kernel build tools installed and the script ran by the
 prowjob included. A couple peculiarities of building/running UML in a docker
 container are solved in the deployment configuration discussed below under "Prow
 Job Specification".

 To build and push, you can either use Bazel or Docker:

 ### Bazel

 Bazel allows building/pushing without dependence on docker and automates
 using latest kunitconfig in build. As Bazel enforces deterministic
 builds, all debian packages depended on are listed in [debs.bzl](debs.bzl), and
 based off the snapshot specified in the *dpkg_src* rule in
 [WORKSPACE](WORKSPACE).

 #### Update

 The direct dependencies needed are:
 * build-essential
 * bc
 * m4
 * flex
 * bison
 * python3

 If recursive dependencies change in future snapshots, [debs.bzl](debs.bzl) must
 be manually updated with a list of all dependencies.
 A simple solution:

 ```shell
 # run interactive shell in latest debian image
 docker run -it --rm debian

 # update list of available packages
 apt update

 # generate list of all required dependencies recursively
 apt-cache depends --recurse --no-recommends --no-suggests --no-conflicts \
 --no-breaks --no-replaces --no-enhances --no-pre-depends \
 build-essential bc m4 flex bison python3 | grep "^\w" | sort -u
 ```

 The snapshot used can be updated as specified in the
 [distroless package manager bazel rules](
 https://github.com/GoogleContainerTools/distroless/tree/master/package_manager).

 #### Use

 ```shell
 # build prowjob image
 bazel build :kunit

 # push to configured repo
 bazel run :push_kunit

 # build test image with kunit source and add to local docker
 bazel run :kunit_test

 # need docker to run, results to stdout
 docker run --privileged --tmpfs /dev/shm:exec kunit-presubmit/test:kunit_test
 ```

 ### Docker

 You must have [Docker](https://docs.docker.com/install/linux/docker-ce/debian/)
 Make sure to enable
 [sudoless docker](https://docs.docker.com/install/linux/linux-postinstall/)
 (fixes gcr push
 authentication problems).

 Now to build and push image :

 ```shell
 # build image
 docker build . gcr.io/kunit-presubmit/kunit

 # confirm image is built
 docker images

 # push to gcr
 docker push gcr.io/kunit-presubmit/kunit
 ```

 To test the container locally:

 ```shell
 # have tmp directory with source checked out to $TMP/linux
 cp Dockerfile.test $TMP/Dockerfile
 cd $TMP

 # build test container which includes source
 docker build . -t test

 # run with args to handle issues with UML in Docker.
 docker run --privileged --tmpfs /dev/shm:exec test

 # extract log
 RUN=$(docker container ls --last 1 -q)
 docker cp $RUN:/artifacts/kunit.log .
 cat kunit.log

 # cleanup container
 docker rm $RUN
 ```

 Note: Testing with an interactive shell results in unexpected behaviour. Running
 the UML Kernel in an entrypoint script works as intended but fails in an
 interactive shell.

 ### Registry

 By default, we have configured our install to use Google Container Registry.
 This requires installing [Google Cloud SDK](
 https://cloud.google.com/sdk/install) and configuring with project to push to.
 If using docker to push, you will need to enable gcloud authentication first to
 push with `gcloud auth configure-docker`.

 ## Prow Job Specification

 Prow Jobs are detailed at
 [testinfra/prow/jobs.md](https://github.com/kubernetes/test-infra/blob/master/prow/jobs.md).
 The prow job specification is held in the
 [config.yaml](config.yaml). It
 specifies the gerrit repo for prow to poll and the specifications for the
 container. We found that running the container privileged and mounting a
 emptydir at /dev/shm fixed KUnit build errors. There may be more secure methods,
 but as prow doesn't expose job containers to external resources, this solution
 results in the cleanest Dockerfile. We are using the decorated prow job which is
 recommended for all new prowjobs and detailed at
 [test-infra/prow/pod-utilities.md](https://github.com/kubernetes/test-infra/blob/master/prow/pod-utilities.md).

 ## Job Script

 The job script and kunitconfig stored inside in the docker image are
 [kunit.sh](kunit.sh) and
 [kunitconfig](kunitconfig).

 Prow sets the environment variable ARTIFACTS to specify a directory that will be
 exported to gcloud on job completion. It additionally sends all data send to
 stdout/stderr to the reporting interface. Exit code of 0 signals success and 1
 signals failure akin to regular shell scripts.

 ## Prow Cluster Deployment

 To deploy in any Kubernetes environment, first read Prow deployment
 documentation [here](https://github.com/kubernetes/test-infra/blob/master/prow/\
 getting_started_deploy.md) and for further clarification, the Kubernetes
 documentation [here](https://kubernetes.io/docs/setup/). Prow comes with several
 [components](https://github.com/kubernetes/test-infra/blob/master/prow/cmd/README.md),
 several of which are only necessary for interacting with github webhooks. We
 have included here a [deployment.yaml](deployment.yaml) which includes just the
 components needed to poll gerrit repos. For every new gerrit repo to run
 presubmits on, you will need to update the
 [Gerrit](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/gerrit)
 component in the deployment accordingly.

 Checkout [prow/cluster/starter.yaml](
 https://github.com/kubernetes/test-infra/blob/master/prow/cluster/starter.yaml)
 for github presubmit instructions which requires oauth token authentication.
 Job configuration in config.yaml does not need to change.

 Succinctly, a Kubernetes deployment file specifies all the API objects needed
 for deployment. Each prow component is a container pod that is described in
 an object of kind *Deployment*, specifying a Docker image that the deployment
 will launch. *Services* specify extra-pod communication and
 *ServiceAccount*s provide an identity to the processes that run in these pods.
 *Role*s and *RoleBinding*s provide RBAC authorization for components. Read this
 [article](https://www.cncf.io/blog/2018/08/01/demystifying-rbac-in-kubernetes/)
 on RBAC in Kubernetes if documentation is not sufficient.
 *PersistentVolumeClaim* are a type of Volume used for maintaining state and
 is used for gerrit to keep track of the latest synced commit. You may also need
 to configure the *Ingress* depending on your network / deployment environment
 for all external communication.

 You will also need a git https cookie file to interact with your gerrit
 instances as anonymous access is not currently supported. For a token
 periodically authenticated with gcloud, see/deploy [grandmatriarch](
 https://github.com/kubernetes/test-infra/blob/master/prow/cmd/grandmatriarch/bake.sh)
 (TODO(avikr) BUG #120081009: change documentation upon gcr image update)
	# Prow Presubmit for KUnit

	KUnit uses Prow for presubmit and CI.

	This repository contains all the code and instructions needed for generating a
	docker container for building and running KUnit tests on Prow, and then using
	that container with a Prow cluster.

	The official repo for Prow is
	[test-infra](https://github.com/kubernetes/test-infra), Kubernetes' test
	infrastructure. It enables us to run presubmit jobs on changes to our gerrit
	repo against an arbitrary docker container image. We have pushed the docker
	image generated from here to gcr.io/kunit-presubmit/kunit. The image is pulled
	and deployed by our prow cluster, source is pulled into working directory of
	container from
	[kunit source](https://kunit.googlesource.com/linux/kunit/alpha/master), and
	entrypoint script (kunit.sh) is ran. The script copies kunitconfig to working
	directory, runs kunit.py, and sends output to job artifacts that can be viewed
	later. Prow will comment on the Gerrit change with the status of the job
	(successful, a test failed, a test crashed, etc) and link to Prow URL for more
	details. The container is currently configured to require the inclusion of
	kunitconfig in repo. A convenience of this model is that updating the job image
	can be done without change to ProwJob configuration.

	## ProwJob Docker image

	The docker image is configured in the
	[Dockerfile](Dockerfile). It is
	based off of Debian, with kernel build tools installed and the script ran by the
	prowjob included. A couple peculiarities of building/running UML in a docker
	container are solved in the deployment configuration discussed below under "Prow
	Job Specification".

	To build and push, you can either use Bazel or Docker:

	### Bazel

	Bazel allows building/pushing without dependence on docker and automates
	using latest kunitconfig in build. As Bazel enforces deterministic
	builds, all debian packages depended on are listed in [debs.bzl](debs.bzl), and
	based off the snapshot specified in the dpkg_src rule in
	[WORKSPACE](WORKSPACE).

	#### Update

	The direct dependencies needed are:
	* build-essential
	* bc
	* m4
	* flex
	* bison
	* python3

	If recursive dependencies change in future snapshots, [debs.bzl](debs.bzl) must
	be manually updated with a list of all dependencies.
	A simple solution:

	```shell
	# run interactive shell in latest debian image
	docker run -it --rm debian

	# update list of available packages
	apt update

	# generate list of all required dependencies recursively
	apt-cache depends --recurse --no-recommends --no-suggests --no-conflicts \
	--no-breaks --no-replaces --no-enhances --no-pre-depends \
	build-essential bc m4 flex bison python3 \| grep "^\w" \| sort -u
	```

	The snapshot used can be updated as specified in the
	[distroless package manager bazel rules](
	https://github.com/GoogleContainerTools/distroless/tree/master/package_manager).

	#### Use

	```shell
	# build prowjob image
	bazel build :kunit

	# push to configured repo
	bazel run :push_kunit

	# build test image with kunit source and add to local docker
	bazel run :kunit_test

	# need docker to run, results to stdout
	docker run --privileged --tmpfs /dev/shm:exec kunit-presubmit/test:kunit_test
	```

	### Docker

	You must have [Docker](https://docs.docker.com/install/linux/docker-ce/debian/)
	Make sure to enable
	[sudoless docker](https://docs.docker.com/install/linux/linux-postinstall/)
	(fixes gcr push
	authentication problems).

	Now to build and push image :

	```shell
	# build image
	docker build . gcr.io/kunit-presubmit/kunit

	# confirm image is built
	docker images

	# push to gcr
	docker push gcr.io/kunit-presubmit/kunit
	```

	To test the container locally:

	```shell
	# have tmp directory with source checked out to $TMP/linux
	cp Dockerfile.test $TMP/Dockerfile
	cd $TMP

	# build test container which includes source
	docker build . -t test

	# run with args to handle issues with UML in Docker.
	docker run --privileged --tmpfs /dev/shm:exec test

	# extract log
	RUN=$(docker container ls --last 1 -q)
	docker cp $RUN:/artifacts/kunit.log .
	cat kunit.log

	# cleanup container
	docker rm $RUN
	```

	Note: Testing with an interactive shell results in unexpected behaviour. Running
	the UML Kernel in an entrypoint script works as intended but fails in an
	interactive shell.

	### Registry

	By default, we have configured our install to use Google Container Registry.
	This requires installing [Google Cloud SDK](
	https://cloud.google.com/sdk/install) and configuring with project to push to.
	If using docker to push, you will need to enable gcloud authentication first to
	push with `gcloud auth configure-docker`.

	## Prow Job Specification

	Prow Jobs are detailed at
	[testinfra/prow/jobs.md](https://github.com/kubernetes/test-infra/blob/master/prow/jobs.md).
	The prow job specification is held in the
	[config.yaml](config.yaml). It
	specifies the gerrit repo for prow to poll and the specifications for the
	container. We found that running the container privileged and mounting a
	emptydir at /dev/shm fixed KUnit build errors. There may be more secure methods,
	but as prow doesn't expose job containers to external resources, this solution
	results in the cleanest Dockerfile. We are using the decorated prow job which is
	recommended for all new prowjobs and detailed at
	[test-infra/prow/pod-utilities.md](https://github.com/kubernetes/test-infra/blob/master/prow/pod-utilities.md).

	## Job Script

	The job script and kunitconfig stored inside in the docker image are
	[kunit.sh](kunit.sh) and
	[kunitconfig](kunitconfig).

	Prow sets the environment variable ARTIFACTS to specify a directory that will be
	exported to gcloud on job completion. It additionally sends all data send to
	stdout/stderr to the reporting interface. Exit code of 0 signals success and 1
	signals failure akin to regular shell scripts.

	## Prow Cluster Deployment

	To deploy in any Kubernetes environment, first read Prow deployment
	documentation [here](https://github.com/kubernetes/test-infra/blob/master/prow/\
	getting_started_deploy.md) and for further clarification, the Kubernetes
	documentation [here](https://kubernetes.io/docs/setup/). Prow comes with several
	[components](https://github.com/kubernetes/test-infra/blob/master/prow/cmd/README.md),
	several of which are only necessary for interacting with github webhooks. We
	have included here a [deployment.yaml](deployment.yaml) which includes just the
	components needed to poll gerrit repos. For every new gerrit repo to run
	presubmits on, you will need to update the
	[Gerrit](https://github.com/kubernetes/test-infra/tree/master/prow/cmd/gerrit)
	component in the deployment accordingly.

	Checkout [prow/cluster/starter.yaml](
	https://github.com/kubernetes/test-infra/blob/master/prow/cluster/starter.yaml)
	for github presubmit instructions which requires oauth token authentication.
	Job configuration in config.yaml does not need to change.

	Succinctly, a Kubernetes deployment file specifies all the API objects needed
	for deployment. Each prow component is a container pod that is described in
	an object of kind Deployment, specifying a Docker image that the deployment
	will launch. Services specify extra-pod communication and
	ServiceAccounts provide an identity to the processes that run in these pods.
	Roles and RoleBindings provide RBAC authorization for components. Read this
	[article](https://www.cncf.io/blog/2018/08/01/demystifying-rbac-in-kubernetes/)
	on RBAC in Kubernetes if documentation is not sufficient.
	PersistentVolumeClaim are a type of Volume used for maintaining state and
	is used for gerrit to keep track of the latest synced commit. You may also need
	to configure the Ingress depending on your network / deployment environment
	for all external communication.

	You will also need a git https cookie file to interact with your gerrit
	instances as anonymous access is not currently supported. For a token
	periodically authenticated with gcloud, see/deploy [grandmatriarch](
	https://github.com/kubernetes/test-infra/blob/master/prow/cmd/grandmatriarch/bake.sh)
	(TODO(avikr) BUG #120081009: change documentation upon gcr image update)