mocking.rst: add start of write-up on mocking + function redirection
Note: this isn't so much about mocking as it is about redirecting
function calls (potentially to a mock). But kunit.dev/mocking will
probably be more memorable and match more what people search for.
In [1], a doc was presented on stream but couldn't be made publicly
available.
This is the start of the process of making it available.
Unlike that doc, this page has a different audience and is intended to
also try and give a basic feel for what mocking is and why you
would/would not want to use it.
This first patch includes the description of fakes vs mocks and a
skeleton for the rest (the doc's overview of the different kinds of
approaches to adding indirection).
[1] https://events.linuxfoundation.org/mentorship-session-kunit-testing-strategies/
Change-Id: I54899388cc2ca07b1ba5d41dc7add4363cab2223
Signed-off-by: Daniel Latypov <dlatypov@google.com>
diff --git a/index.rst b/index.rst
index 12194d6..2936d33 100644
--- a/index.rst
+++ b/index.rst
@@ -15,6 +15,7 @@
development/index
third_party/kernel/index.rst
third_party/stable_kernel/index.rst
+ mocking
press
What is KUnit?
diff --git a/mocking.rst b/mocking.rst
new file mode 100644
index 0000000..b84bbe1
--- /dev/null
+++ b/mocking.rst
@@ -0,0 +1,243 @@
+====================================
+Fakes and Stubbing and Mocks, Oh My!
+====================================
+
+This page seeks to provide an overview on mocking and a related task:
+redirecting function calls to test-only code. Note: many people use the term
+"mocking" to refer to the latter (and that's fine!), but we'll try and keep the
+concepts separate in this doc.
+
+KUnit currently lacks specific support for either of these, in part due to the
+fact there's enough trade-offs that it's hard to come up with a generic
+solution.
+
+Why do we need this?
+====================
+
+First, let's consider what the goal is. We want unit tests to be as
+lightweight and hermetic as possible, and only test the code we care about.
+
+A canonical example in userspace testing to consider is a database.
+We'd want to verify that our code behaves properly (inserts the right rows to
+the database, etc.), but we don't want to bring up a test database every time
+we run our tests.
+
+Not only will this make the test take longer to run, it also adds more
+opportunities for the test to break in uninteresting ways, e.g. if writes to
+the database fail due to transient network issues.
+
+If we can construct a "fake" database that implements the same interface, which
+is simply an in-memory hashtable or array, then we can have much faster and
+more reliable tests. Unit tests simply don't need the scability and features of
+a real database.
+
+Fakes versus mocks
+==================
+
+We'll be using terminology roughly as defined in
+https://martinfowler.com/bliki/TestDouble.html, namely:
+
+- a "test double" is the more generic term for any kind of test-only replacement.
+- a "mock" is a test double that specifically can make assertions about how its
+ called and can return different values based on its inputs.
+- a "fake" is a test double that mimics the semantics of the code it's replacing
+ but with less overhead and dependencies, e.g. a fake database might just use
+ a hash table, or a fake IO device which is just a ``char buffer[MAX_SIZE]``, or UML itself (in a sense).
+
+| Mocks generally are written with support from their testing framework, whereas fakes are typically written without them.
+| KUnit currently lacks any features to specifically facilitate mocks, so it's recommended to create and use fakes.
+
+Downsides of mocking
+--------------------
+
+Very briefly, using mocks in tests can make tests more fragile since they test
+"behavior" rather than "state."
+
+What do we mean by that? Let's imagine we're testing some userspace program
+with gMock-like syntax (a C++ mocking framework):
+
+.. code-block:: c
+
+ void send_data(struct data_sink *sink)
+ {
+ /* do some fancy calculation to figure out what to write */
+ sink->write("hello, ");
+ sink->write("world");
+ }
+
+ void test_send_data(struct test *test)
+ {
+ struct data_sink *sink = make_mock_datasink();
+
+ EXPECT_CALL(data_sink, write("hello, "))
+ .WillOnce(Return(7));
+ EXPECT_CALL(data_sink, write("world"))
+ .WillOnce(Return(5));
+ send_data(sink);
+ }
+
+And now let's say we've realized we can make our code twice as fast with more
+buffering, effectively changing it to:
+
+.. code-block:: c
+
+ void send_data(struct data_sink *sink)
+ {
+ sink->write("hello, world");
+ }
+
+
+| Oops, now our mock-based tests are failing since we've changed how many times we call ``write()``!
+| Contrast this to a state-based approach where ``write()`` might just append to some ``char buffer[MAX_SIZE]``. In that case, we can validate ``send_data()`` worked by just using ``KUNIT_EXPECT_STREQ(test, buffer, "hello, world")`` and it would work for either implementation.
+
+A further downside is that the test author has to mimic the behavior
+themselves, i.e. the return values for each ``write()`` call. This means if
+the test author makes a mistake or tests just don't get updated after a
+refactor, the mock can behave in unrealistic fashion.
+
+This can and *will* eventually lead to bugs.
+
+
+Upsides of mocking
+------------------
+
+| This is not to say that one should never test "behaviour", i.e. use mocking.
+| E.g. imagine we *wanted* the example test to validate that we only call ``write()`` once since each call is super-expensive.
+| Or consider when there's no easy way to validate that the state has changed, e.g. if we want to validate that ``prefetchw()`` is called to pull a specific data structure into cache.
+
+
+| It's also easier easier to use a mock if we want to force a certain return value, e.g. if we want to make a specific ``write()`` call fail so we can test an error path.
+| With our ``data_sink`` example above, it's hard for an append into a ``char buffer[MAX_SIZE]`` to fail until we hit ``MAX_SIZE``, but for real code that might be writing to disk or sending data over the network, failure could happen for ~any call. And it's valuable to test that our code is robust against such failures.
+
+Function redirection
+====================
+
+| Regardless of what kind of test double you use, they're useless unless you can swap out the real code for them.
+| For lack of a better term, we'll refer to this as function redirection: how do I make calls to ``real_function()`` go to my ``fake_function()``?
+
+| In other test frameworks (Python's unittest, JUnit for Java, Googletest for C++, etc.), this is fairly easy. This is because they rely on techniques like dynamic dispatch, which has language support.
+| We can and do re-implement dynamic dispatch in the kernel in C, but this adds runtime overhead which may or may not be acceptable in all contexts.
+
+The problem boils down to `adding another layer of indirection
+<https://en.wikipedia.org/wiki/Fundamental_theorem_of_software_engineering>`_
+and we have various options to choose from, which we'll describe below.
+
+For each of these, let's consider the following code:
+
+.. code-block:: c
+
+ static void func_under_test(void)
+ {
+ /* unsafe to call this function directly in a test! */
+ send_data_to_hardware("hello, world\n");
+ }
+
+Run time (ops structs, "class mocking")
+---------------------------------------
+
+This is the most straightforward approach and fundamentally boils down to doing
+this:
+
+.. code-block:: c
+
+ static void func_under_test(void (*send_data_func)(const char *str))
+ {
+ send_data_func("hello, world\n");
+ }
+
+
+Being a bit more sophisticated, we can introduce a struct to hold the
+functions:
+
+.. code-block:: c
+
+ struct send_ops {
+ void (*send)(const char *str);
+ /* maybe more functions here in real code */
+ };
+
+TODO(dlatypov@google.com): write about "class mocking", `RFC here
+<https://lore.kernel.org/linux-kselftest/20201012222050.999431-1-dlatypov@google.com/>`_
+
+Pros:
+~~~~~
+
+- Simplest implementation: "it's just code."
+- This is the only approach here where we can limit the scope of the
+ redirection.
+
+ - The subsequent approaches **globally** redirect all calls to
+ ``send_data_to_hardware()``, potentially in code not-under-test we
+ don’t want to mess with.
+- There are plenty of such structs throughout the kernel.
+
+ - And users don't need any special support from KUnit.
+
+Cons:
+~~~~~
+
+- ~Everyone knows about this convention but still want "mocking." It's not seen
+ as sufficient by itself.
+- Requires the most invasive code changes if the code isn't already using this
+ pattern.
+
+ - Introduces runtime overhead (an indirect call, another function
+ argument, etc.)
+- If ``func_under_test()`` is publicly exposed, but ``send_data_func()`` is not
+ (most likely the case), users need to workaround this.
+- The `RFC for "class mocking"
+ <https://lore.kernel.org/linux-kselftest/20201012222050.999431-1-dlatypov@google.com/>`_
+ requires a lot of boilerplate, even after providing macros to take care of
+ most of it.
+
+ - This is fundamentally a limitation of C (as opposed to C++ where
+ classes have language support). It’s unlikely we can improve much
+ here.
+
+Compile time
+------------
+
+TODO(dlatypov@google.com): write me
+
+Pros:
+~~~~~
+
+- TODO
+
+Cons:
+~~~~~
+
+- TODO
+
+Link time (__weak symbols)
+--------------------------
+
+TODO(dlatypov@google.com): write me
+
+Pros:
+~~~~~
+
+- TODO
+
+Cons:
+~~~~~
+
+- TODO
+
+Binary-level (ftrace et. al)
+----------------------------
+
+TODO(dlatypov@google.com): write me
+
+Pros:
+~~~~~
+
+- TODO
+
+Cons:
+~~~~~
+
+- TODO
+
+TODO(dlatypov@google.com): include discussion on global functions/general statefulness.
+TODO(dlatypov@google.com): include section on worked example use cases.
