Added most of the development documentation
Added documentation on how to contribute, design, gerrit, and prow.
Change-Id: I4a78bab3e14804cd0768dd1f2178543499962d3c
Signed-off-by: Brendan Higgins <brendanhiggins@google.com>
diff --git a/development/design.rst b/development/design.rst
new file mode 100644
index 0000000..ae166d9
--- /dev/null
+++ b/development/design.rst
@@ -0,0 +1,121 @@
+============
+KUnit Design
+============
+
+High Level Structure
+====================
+
+KUnit in practice is divided into two components: the KUnit kernel (more often
+referred to as the KUnit kernel, or just the kernel), and the kunit_tool. Most
+of KUnit, including all the test cases, all of the test libraries, and even the
+test runner itself are all part of the Linux kernel.
+
+.. todo: Discuss the non-UML design of KUnit.
+
+.. _uml:
+
+User Mode Linux
+---------------
+
+User Mode Linux, or UML, is a way to compile and run the Linux kernel as a
+normal program that can run in userspace without the help of a VM or any
+virtualization support.
+
+UML is an architecture `arch/um/
+<https://elixir.bootlin.com/linux/latest/source/arch/um>`_ that maps all the
+architecture specific elements, which would normally map to low level hardware
+features, to the POSIX interface allowing the Linux kernel to be compiled and
+run as a normal userspace program.
+
+For more information on UML, see: http://user-mode-linux.sourceforge.net/
+
+KUnit Kernel
+------------
+
+Most of the complexity of KUnit lies in the Linux kernel itself. Within the
+KUnit kernel there are several parts:
+
+- The :ref:`test-runner` which schedules and runs all the tests, which is
+ composed of:
+
+ - The Linux initcall subsystem which invokes:
+ - The :ref:`test-case-runner` which handles sandboxing and running individual
+ test cases.
+
+- The :ref:`test-library` which provide wrappers around the API provided by the
+ Test Case Runner to test cases to communicate with it.
+
+The Linux kernel itself is **much** more complicated than this, but a full
+overview is outside of the scope of this document.
+
+.. _test-runner:
+
+Test Runner
+~~~~~~~~~~~
+
+The test runner in KUnit is more or less the KUnit kernel itself. Each `kunit
+suite <../third_party/kernel/docs/api/test.html#c.kunit_suite>`_, KUnit's name
+for a test suite, registers its test case runner as an init call with the Linux
+initcall system with |kunit_test_suite|_. When the kernel boots up, it runs
+through all the initcalls to initialize all its subsystems; the last initcalls
+it executes are the test case runners.
+
+.. |kunit_test_suite| replace:: ``kunit_test_suite(suite)``
+.. _kunit_test_suite: third_party/kernel/docs/api/test.html#c.kunit_test_suite
+
+.. _test-case-runner:
+
+Test Case Runner
+~~~~~~~~~~~~~~~~
+
+Each test case runner is associated with a `kunit suite
+<../third_party/kernel/docs/api/test.html#c.kunit_suite>`_. When the test case
+runner is invoked, it runs through each test case and for each test case, does
+the following:
+
+- Register a failure handler with the UML trap subsytem.
+- Run the init function provided by the ``kunit_suite``.
+- Run the test case.
+- Run the exit function provided by the ``kunit_suite``.
+- Deallocate all the test allocated resources.
+- Report test results.
+
+All this happens in `lib/kunit/test.c
+<https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/tree/lib/kunit/test.c?h=test&id=741a98d022362c90609ac9dcd8ad56314d8e68b3>`_.
+
+.. _test-library:
+
+Test Library
+~~~~~~~~~~~~
+
+Currently, the test runner provides the function ``kunit_do_assertion`` for the
+test case to communicate a failed assertion:
+
+.. code-block:: c
+
+ void kunit_do_assertion(struct kunit *test,
+ struct kunit_assert *assert,
+ bool pass,
+ const char *fmt, ...);
+
+It takes a
+
+- `kunit (test context object) <../third_party/kernel/docs/api/test.html#c.kunit>`_
+- `assertion object <https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/tree/include/kunit/assert.h?h=test&id=741a98d022362c90609ac9dcd8ad56314d8e68b3#n42>`_
+- whether the expectation/assertion failed or not
+- log message to show on failure represented as a C format string.
+
+The function definition can be found here:
+https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/tree/lib/kunit/test.c?h=test&id=741a98d022362c90609ac9dcd8ad56314d8e68b3#n152
+
+kunit_tool
+----------
+
+Currently the main purpose kunit_tool serves is just to make to process of
+building the kernel and running tests easier, but it also serves as a place
+where workarounds can be added in. For example, there are some bugs (for
+example, if the UML kernel crashes, it will corrupt the user's terminal
+settings) with the UML user interface that are much easier to work around
+outside of the kernel than fix in the kernel itself. One of the function it
+serves is to parse the kernel output and print out the summarized results in a
+nice, human readable format.
diff --git a/development/gerrit-workflow.rst b/development/gerrit-workflow.rst
new file mode 100644
index 0000000..8ac71a9
--- /dev/null
+++ b/development/gerrit-workflow.rst
@@ -0,0 +1,183 @@
+===============
+Gerrit Workflow
+===============
+
+KUnit can refer both to the unit testing code within the Linux kernel and to
+the project as a whole, which includes some other supporting code. The meaning
+is usually clear when used in context.
+
+With that said, KUnit, as a project, is made up of several repositories. As
+mentioned elsewhere, the actual KUnit unit testing code is in the upstream
+Linux kernel or is otherwise staged on Gerrit waiting to get into the Linux
+kernel. All our other code is in Gerrit. For an introduction to Gerrit (with a
+high level overview of git), see
+https://gerrit-review.googlesource.com/Documentation/intro-user.html. For
+detailed introduction to git, see https://try.github.io.
+
+Starting from Scratch
+=====================
+
+This section describes how to start a change from scratch.
+
+Cloning a repo for the first time
+---------------------------------
+
+If you have not done so yet, clone the repository you want to make changes to.
+For example:
+
+.. code-block:: bash
+
+ git clone https://kunit.googlesource.com/prow-presubmit
+
+Checkout the desired branch
+---------------------------
+
+Checkout the branch on which you want to make changes to. For example:
+
+.. code-block:: bash
+
+ git checkout master
+
+Make sure your branch is up to date
+-----------------------------------
+
+If you have not done so in a while, you will need to update your repo and
+branches. Your remote references can be updated with the ``fetch`` command. For
+example:
+
+.. code-block:: bash
+
+ git fetch origin
+
+where ``origin`` it the name of the KUnit remote, by default ``origin`` is the
+name of the repository that was cloned.
+
+Now that you have updated your remote references, you need to update the local
+branch you will be working on. If this branch is only local to your repository,
+you may need to rebase it on the remote branch you are trying to get your
+changes into (in this case, you would already have changes underway. See
+:ref:`updating-changes` for more information.); here we assume that is not the case.
+Update your branch with the `pull` command. For example:
+
+.. code-block:: bash
+
+ git pull origin -ff
+
+``-ff`` forces git not to merge upstream changes into your branch. This is
+important as we, the KUnit project, do not want merge commits in our git repos.
+This command will fail if you have local changes.
+
+Making a change
+---------------
+
+Once you have made a change, that you are happy with and would like to send up
+for review you first need to ``commit`` the change. For example:
+
+.. code-block:: bash
+
+ git add path/to/a/file/you/modified
+ git add path/to/a/different/file/you/changed
+ git commit
+
+A git commit should be a logical, bite-size change that is easy for your
+reviewer to understand.
+
+Your commit message should have a:
+
+- **subject** - line on top that provides a vague idea of what you are doing.
+- **body** - which describes in more detail what you are doing.
+
+The bottome of your commit message should have:
+
+- ``Change-Id`` - see
+ https://gerrit-review.googlesource.com/Documentation/user-changeid.html for
+ more information.
+- ``Signed-off-by`` - see
+ https://www.kernel.org/doc/html/latest/process/submitting-patches.html#sign-your-work-the-developer-s-certificate-of-origin
+
+Sending changes for review
+--------------------------
+
+Now that you are happy with your commits, you need to send them for review.
+This is done by pushing them to Gerrit with the ``push`` command. For example,
+let's say you want your changes to be submitted on ``master``
+branch, and ``origin`` is the prow-presubmit repo:
+
+.. code-block:: bash
+
+ git push -u origin HEAD:refs/for/master
+
+If pushing succeeds, Gerrit will respond by printing out the link where you can find the review.
+
+.. _updating-changes:
+
+Updating Existing Changes
+=========================
+
+This section describes how to update your or another user's changes.
+
+Checking out someone else's changes
+-----------------------------------
+
+Gerrit allows users to take over changes made by other users. This can be done
+by checking out a change that is currently under review (and then pushing
+modified versions of those changes). Let's say there is a change at
+https://kunit-review.googlesource.com/c/prow-presubmit/+/2569; you can checkout
+this change and its dependencies by clicking on the download button; it will
+provide you several options, the first option provides a git command that you
+can copy to your command line and run, such as:
+
+.. code-block:: bash
+
+ git fetch "https://kunit.googlesource.com/prow-presubmit" refs/changes/69/2569/1 && git checkout FETCH_HEAD
+
+.. tip::
+ You can also use this to checkout changes that you have misplaced or lost.
+
+Updating changes
+----------------
+
+Once you have checked out the changes that you need to update, you may want to
+update them, as the branch you are looking to submit to may have progressed;
+this can be accomplished by rebasing against the branch you want to submit to.
+For example:
+
+.. code-block:: bash
+
+ git rebase master
+
+Now you need to modify the commits to address reviewer's comments, or maybe you
+realized that you forgot something that should be in an earlier commit.
+Modifying commits may be accomplished with interactive rebase. Interactive
+rebase is a very powerful git command and is discussed in some detail here:
+https://dev.to/blakedeboer/beginners-guide-to-interactive-rebasing-1ob.
+
+You might start an interactive rebase with:
+
+.. code-block:: bash
+
+ git rebase -i master
+
+.. note::
+ Only modify commits that are actually under review; changing commits that
+ have already been checked in will cause Gerrit to yell at you.
+
+Sending changes for review (again)
+----------------------------------
+
+All you need to do to update your changes on Gerrit under review is to push
+your modified commits to the same branch that you would have as if you were
+pushing for the first time.
+
+For example, let's say you want to submit your changes to ``master``; you would
+then
+
+.. code-block:: bash
+
+ git push -u origin HEAD:refs/for/master
+
+You may wonder how Gerrit knows to replace the old commits at this location
+with the ones that you modified; the secret is the ``Change-Id``. **A Change-Id
+should almost never be changed when updating a commit**; it tells Gerrit that
+the commit is the same commit as before even though everything else may have
+changed about the commit (including the commit hash).
diff --git a/development/index.rst b/development/index.rst
index 3ac14d2..be63c76 100644
--- a/development/index.rst
+++ b/development/index.rst
@@ -2,6 +2,16 @@
Development
===========
+.. toctree::
+ :maxdepth: 2
+ :caption: Contents:
+
+ start
+ gerrit-workflow
+ design
+ useful-commands
+ prow
+
This set of pages provides information for developing KUnit itself. These pages
are for you if you plan on making submissions to KUnit, you are using KUnit for
really advanced use cases, or you just want to know more about how KUnit works
@@ -9,10 +19,10 @@
Checkout:
-.. TODO(brendanhiggins): add pages for each of these topics.
-
-- Getting Started for getting developing for KUnit.
-- Gerrit Workflow to see how you might structure your workflow to work with git and Gerrit.
-- KUnit Design to learn about how KUnit is organized and how it got here.
-- Useful Commands for a collection of tips and tricks while working on KUnit.
-- Release Processes on how to cut a release for KUnit.
+- :doc:`start` for getting developing for KUnit.
+- :doc:`gerrit-workflow` to see how you might structure your workflow to work
+ with git and Gerrit.
+- :doc:`design` to learn about how KUnit is organized and how it got here.
+- :doc:`useful-commands` for a collection of tips and tricks while working on
+ KUnit.
+- :doc:`prow` for information on the CI/CD system we use.
diff --git a/development/prow.rst b/development/prow.rst
new file mode 100644
index 0000000..ce02967
--- /dev/null
+++ b/development/prow.rst
@@ -0,0 +1,18 @@
+===============================
+Prow Presubmit for KUnit Gerrit
+===============================
+
+We, KUnit, use `Prow
+<https://github.com/kubernetes/test-infra/tree/master/prow>`_, a Kubernetes
+based CI/CD system, for our Presubmit system.
+
+Prow has built in support for Gerrit and GitHub integration. The only thing
+needed to run KUnit on Prow is a special test runner container, which we
+maintain here: https://kunit.googlesource.com/prow-presubmit.
+
+The Prow config which specifies our container can be found `here
+<https://github.com/GoogleCloudPlatform/oss-test-infra/blob/master/prow/prowjobs/kunit-review.googlesource.com/linux/kunit-linux-config.yaml>`_
+in the google-oss prow cluster.
+
+For more information, please refer to our Prow container repo:
+https://kunit.googlesource.com/prow-presubmit/
diff --git a/development/start.rst b/development/start.rst
new file mode 100644
index 0000000..71936c4
--- /dev/null
+++ b/development/start.rst
@@ -0,0 +1,80 @@
+================================
+Getting Started Developing KUnit
+================================
+
+Setting Up Your Work Environment
+================================
+
+Follow the instructions at `../usage/index`.
+
+Familiarize Yourself with KUnit's Design
+========================================
+
+KUnit's design is discussed in the :doc:`design` section.
+
+Contributing to code in the Linux kernel
+========================================
+
+Please follow the guide in the Linux kernel:
+https://www.kernel.org/doc/html/latest/#introduction-to-kernel-development
+
+In addition, remember to CC kunit-dev@googlegroups.com and
+linux-kselftest@vger.kernel.org.
+
+Contributing to code outside of the Linux kernel
+================================================
+
+This section is for contributing code to repos that live outside of
+https://git.kernel.org.
+
+Why is there a distinction?
+---------------------------
+
+The Linux kernel has been around a lot longer than KUnit, and thus has rules
+that we have no say on. However, any code that lives outside of the kernel is
+managed by Google; it's open source and everyone has equal right to it, but
+because it is under the Google umbrella and not the Linux Foundation's umbrella
+it means that there are certain rules we have to follow in one place, and
+different rules in the other.
+
+The details are not important, all you have to know is that if you send code to
+the Linux kernel you should follow the rules above, and if you are contributing
+to any other KUnit repos, you should follow the rules here.
+
+If you still have confusion, feel free to reach out to us.
+
+Sign the Contributor License Agreement
+--------------------------------------
+
+For any contributions not sent directly to the Linux kernel via the Linux
+kernel mailing lists, we require contributors to sign our Contributor License
+Agreement (CLA).
+
+You may sign our CLA here: https://cla.developers.google.com/
+
+.. note::
+ Your company may already have a CLA on file with Google.
+
+Submitting Changes
+------------------
+
+We use `Gerrit <https://kunit-review.googlesource.com>`_ for all non-upstream
+code reviews.
+
+In order for Gerrit to accept your change, you must first make sure your commit
+messages are formatted correctly. It should have a:
+
+- ``Change-Id`` - see
+ https://gerrit-review.googlesource.com/Documentation/user-changeid.html for
+ more information.
+- ``Signed-off-by`` - see
+ https://www.kernel.org/doc/html/latest/process/submitting-patches.html#sign-your-work-the-developer-s-certificate-of-origin
+
+Now you are ready to send your change for review:
+
+.. code-block:: bash
+
+ git push -u origin HEAD:refs/for/master
+
+For more information on submitting changes through Gerrit, please see the
+following section :doc:`gerrit-workflow`.
diff --git a/development/useful-commands.rst b/development/useful-commands.rst
new file mode 100644
index 0000000..8f41b56
--- /dev/null
+++ b/development/useful-commands.rst
@@ -0,0 +1,30 @@
+===============
+Useful Commands
+===============
+
+Building and Running KUnit Manually
+===================================
+
+Sometimes for debugging kunit_tool or the interaction between the KUnit kernel
+and kunit_tool it is useful to build and run the tests manually:
+
+.. code-block:: bash
+
+ cp kunitconfig .config
+ make ARCH=um -j 12
+ ./linux mem=1G log_buf_len=2M
+
+Cleaning the Kernel
+===================
+
+As typical, cleaning object files can be accomplished with:
+
+.. code-block:: bash
+
+ make clean
+
+and the tree can be fully cleaned with:
+
+.. code-block:: bash
+
+ make mrproper
