release_notes.rst

=============
Release Notes
=============

This page briefly summarizes the KUnit changes in each Linux release.

For more specific documentation on KUnit features, see the in-kernel
documentation: https://kernel.org/doc/html/latest/dev-tools/kunit.

.. _v5.5:

5.5
===

- KUnit introduced, including all the basic functionality.
- ``kunitconfig`` was renamed to ``.kunitconfig`` later in 5.5 (`commit
  <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=14ee5cfd4512ee3d1e0047d8751450dcc6544070>`__)

.. _v5.6:

5.6
===

- KUnit tests can be built as modules, `docs
  <https://www.kernel.org/doc/html/latest/dev-tools/kunit/running_tips.html#running-tests-as-modules>`__

.. _v5.7:

5.7
===

- kunit.py switched to using ``.kunit/`` as the default build dir (and
  ``.kunitconfig`` => ``.kunit/.kunitconfig``).
- ``CONFIG_KUNIT_DEBUGFS`` to expose per-suite test results in
  ``/sys/kernel/debug/kunit/<suite>/results``, `docs
  <https://www.kernel.org/doc/html/latest/dev-tools/kunit/running_tips.html#retrieving-per-suite-results>`__

.. _v5.8:

5.8
===

- ``kunit.py`` now supports separate ``config/build/exec/parse`` commands in
  addition to ``run``.

  - You can use these if you don't want to let ``kunit.py run`` do everything
    for you, e.g. if you want to run manually but parse the results using
    ``kunit.py``.

- ``CONFIG_KUNIT_ALL_TESTS`` introduced

  - This is just a convention. New tests should use it like so:

.. code-block:: none

	config FOO_KUNIT_TEST
		tristate "KUnit test for foo" if !KUNIT_ALL_TESTS
		depends on KUNIT
		default KUNIT_ALL_TESTS
		help
		  This builds unit tests for foo.


See `the upstream documentation
<https://www.kernel.org/doc/html/latest/dev-tools/kunit/style.html#test-kconfig-entries>`__
for the authoritative source on how to write Kconfig entries.

.. _v5.9:

5.9
===

- Named resources: tests can store and retrieve data via string keys.

  - This is a power feature intended to be a cleaner alternative to using
    global variables in tests.
  - See `test_kasan.c
    <https://elixir.bootlin.com/linux/latest/source/lib/test_kasan.c>`__ for an
    example that uses ``"kasan_data"``.

.. _v5.10:

5.10
====

- KASAN violations cause the current KUnit test case to be marked FAILED.

  - Note: KASAN doesn't currently work on UML.

.. _v5.11:

5.11
====

- Parameterized testing: see the `docs
  <https://www.kernel.org/doc/html/latest/dev-tools/kunit/usage.html#parameterized-testing>`__.

.. _v5.12:

5.12
====

- You can pass in a path to a kunitconfig file instead of using
  ``.kunit/.kunitconfig``, e.g.

.. code-block:: shell

        $ ./tools/testing/kunit/kunit.py run --kunitconfig=fs/ext4/.kunitconfig

- You can tell ``kunit.py`` to run a subset of built suites by passing in a
  glob argument, e.g.

.. code-block:: shell

	# Only run "list" suites
	$ ./tools/testing/kunit/kunit.py run '*list*'

.. _v5.13:

5.13
====

-  Can call ``kunit_fail_current_test()`` from anywhere to fail the current KUnit test (e.g. if an invariant is broken, a UBSAN violation is detected, etc.)

- You can pass a directory ``--kunitconfig`` and it'll implicitly append
  ``.kunitconfig``, e.g. ``kunit.py run --kunitconfig=lib/kunit``.

.. _v5.14:

5.14
====

- ``kunit.py`` can run tests on non-UML architectures (using QEMU), `docs
  <https://www.kernel.org/doc/html/latest/dev-tools/kunit/kunit-tool.html#running-tests-on-qemu>`__
- It's now possible to generate code coverage reports when using UML, `docs
  <https://www.kernel.org/doc/html/latest/dev-tools/kunit/running_tips.html#generating-code-coverage-reports-under-uml>`__
- Tests can call ``kunit_skip()`` to bail out and mark a test as "SKIPPED"
  instead of "PASSED" or "FAILED"

.. _v5.15:

5.15
====

- UBSAN violations cause the current KUnit test case to be marked FAILED.
- Can pass kernel commandline parameters from ``kunit.py`` via
  ``--kernel_args``
- KUnit prints a summary per suite of how many test cases passed/failed to
  dmesg, e.g.

.. code-block:: shell

	# property-entry: pass:7 fail:0 skip:0 total:7

.. _v5.16:

5.16
====

- You can filter on test names now in addition to suite names, e.g.

.. code-block:: shell

	# Only run tests that contain "foo"
	$ ./tools/testing/kunit/kunit.py run '*.*foo*'
	# Only run tests in "mysuite" that contain "foo"
	$ ./tools/testing/kunit/kunit.py run 'mysuite.*foo*'

- You can ask ``kunit.py`` to run each suite/test by itself. This can be useful
  to track down issues due to non-hermetic tests

.. code-block:: shell

	$ ./tools/testing/kunit/kunit.py run --run_isolated=suite
	$ ./tools/testing/kunit/kunit.py run --run_isolated=test

- ``kunit.py``'s parsing logic was largely rewritten, so the output looks
  slightly different

  - Parsed output is also now mostly printed out in real time.

- ``kunit.py`` with ``--raw_output`` will print output in real time.

- In-kernel documentation has been significantly revamped.

.. _v5.17:

5.17
====

- Print out parsed test results in real time.

- Allow tweaking Kconfig options from command-line, e.g.

.. code-block:: shell

	$ ./tools/testing/kunit/kunit.py run --kconfig_add=MY_CONFIG=y

- Reconfigure kunit kernel when the used ``kunitconfig`` has changed

  - Before, if you removed an entry from ``.kunitconfig``, ``kunit.py`` would
    not reconfigure, so your edit would have no effect.

  - This now also works if you remove a ``--kconfig_add`` flag.

- Print out parameters as sub-test cases (both in raw and parsed output), e.g.:

.. code-block::

	[12:37:14] =============== ext4_inode_test (1 subtest) ================
	[12:37:14] ======= inode_test_xtimestamp_decoding (16 subtests) =======
	[12:37:14] [PASSED] 1901-12-13 Lower bound of 32bit < 0 timestamp, no extra bits
	...
	[12:37:14] [PASSED] 2446-05-10 Upper bound of 32bit >=0 timestamp. All extra sec bits on
	[12:37:14] ========= [PASSED] inode_test_xtimestamp_decoding ==========
	[12:37:14] ================= [PASSED] ext4_inode_test =================
	[12:37:14] ============================================================

.. _v5.18:

5.18
====

- Drastically decrease stack usage of ``KUNIT_EXPECT_`` macros.

  - Before, we largely relied on the compiler to optimize away some internal
    structs, but that didn't always work.

  - Now on UML (x86_64), ``KUNIT_EXPECT_EQ()`` uses 32 bytes instead of 88.

.. _v5.19:

5.19
====

- Add support for suite-level init/exit functions.

  - ``suite_init``/``suite_exit`` are like ``init`` and ``exit`` instead of once per test case.

- Add ``KUNIT_EXPECT_NULL()``/``KUNIT_EXPECT_NOT_NULL()`` macros.

- Split out KUnit resource API from ``<kunit/test.h>`` into ``<kunit/resource.h>``.

  - Including ``<kunit/test.h>`` will still automatically ``#include <kunit/resource.h>`` for now.

- ``kunit.py`` no longer colorizes output when stdout is not a tty.

- Add a manually curated ``tools/testing/kunit/configs/all_tests_uml.config``

  - This attempts to enable as many tests as possible on UML.

- Change quoting requirements for qemu_config files (`commit
  <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3f0a50f345f78183f6e9b39c2f45ca5dcaa511ca>`__)

  - E.g. instead of ``['-m 256']``, do ``['-m', '256']``

.. _v6.0:

6.0
===

- Overhaul support for building tests as modules

  - Before, it wouldn't work in modules that already had their own
    ``module_init``/``module_exit``. Now this restriction is gone.

- Add ``--qemu_args`` flag, e.g. use ``--qemu_args='-m 2048'`` to give tests more RAM.

- Allow ``--kunitconfig`` to be repeated.

  - This will just concatenante the contents together.
  - One use case for this is appending the options for code coverage:

.. code-block:: shell

	$ ./tools/testing/kunit/kunit.py run --kunitconfig=my/dir --kunitconfig=tools/testing/kunit/configs/coverage_uml.config

- Taint the kernel if any KUnit tests run.

  - Add a new ``TAINT_TEST`` flag, shows up as ``'N'``.

- Add support for KASAN to run on 64-bit UML.

- By default, enable PCI support under UML.

.. _v6.1:

6.1
===

- ``kunit.py run --alltests`` now works.

  - Renamed ``all_tests_uml.config`` (added in :ref:`v5.19`) to ``all_tests.config``.

  - Changed the ``--alltests`` flag to use this manually curated file instead of
    trying to get ``allyesconfig`` to work.

  - Removed Virtio and PCI UML specific options from ``all_tests_uml.config``.

- ``Documentation/`` updates and fixes.

- Further reduction in the stack consumption of ``KUNIT_EXPECT_`` macros.

  - For context, see :ref:`v5.18`.

  - On UML, now ``KUNIT_FAIL()`` uses 0 bytes (was 8), and ``KUNIT_EXPECT_EQ()``
    uses 24 (was 32).

- In edge cases, ``kunit_kfree()`` matches ``kfree()``:

  -  ``kunit_kfree(NULL)`` is a no-op.

  -  ``kunit_kfree(invalid_ptr)`` will no longer crash, but it will now fail the test.

- Introduce new ``kunit.enable`` kernel command-line argument.

  - This is a niche feature. If you don't want to run tests, prefer setting
    ``CONFIG_KUNIT=n``.

  - However, if you need to compile KUnit, you can use this to forcibly prevent
    tests from running.

  - You can set the default value of the option via ``CONFIG_KUNIT_DEFAULT_ENABLED``.

.. _v6.2:
6.2
===

- Added new ``KUNIT_EXPECT_MEMEQ(test, a, b, size)`` macro for comparing memory blocks.

  - This is an alternative to ``KUNIT_EXPECT_EQ(test, memcmp(a, b, size), 0)``.

  - On failure, KUnit will print out the bytes and highlight where the regions differ.

- Failure logs are available in decimal and hexadecimal output.

- Use a static key to check if tests are currently running.

  - This reduces the performance overhead of `kunit_get_current_test()` and
    `kunit_fail_current_test()` when no test is running.

  - For more details, see `kunit: Provide a static key to check if KUnit is actively running tests <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=908d0c177bbc7c34ab9129c6f2bcd87487115632>`__.

- Make KUnit output and the kunit.py parser compliant with KTAP version 1 specification.

  - In KUnit output, subtests are now indicated using "KTAP version 1" headers.

- Added two macros, `VISIBLE_IF_KUNIT` and `EXPORT_SYMBOL_IF_KUNIT`, that let
  tests to conditionally expose  static symbols.

  - For an example of how to use these macros, see `apparmor: test: make static symbols visible during kunit testing <https://git.kernel.org/pub/scm/ linux/kernel/git/torvalds/linux.git/commit/?id=b11e51dd70947107fa4076c6286dce301671afc1>`__.

- Made the following KUnit documentation changes:

  - Updated the "architecture.rst" page for style and grammatical issues.

  - Made "usage.rst" a superset of "tips.rst" and removed "tips.rst" page.

  - Reworded assertion description.

  - Fix "How Do I Use This" / "Next Steps" section.

  - Updated the requirements for test result description in KTAP specification.

.. _v6.3:

6.3
===

- Added "hooks" to call into KUnit when it's built as a module.

  - These allow some KUnit features to be used in built-in code, or in other
    modules, without introducing a hard dependency on KUnit.

- Introduced a "static_stub" feature to add redirection support for KUnit tests.
  Any function that wants to be intercepted adds a call to a macro which checks
  if a test has redirected calls to it, and then calls the corresponding replacement.

.. _v6.4:

6.4
===

- kunit.py now supports building / running tests via QEMU on the m68k and SuperH (sh)
  architectures.

- Several fixes to debugfs log support, particularly with parameterized tests.

- The default log size has been increased to 2048 bytes.

.. _v6.5:

6.5
===

- Added a new 'deferred actions' API, based on the devm\_/devres APIs, to call a function
  automatically on test exit. This should simplify resource management an cleanup.

  - See: https://docs.kernel.org/dev-tools/kunit/usage.html#registering-cleanup-actions

- Test cleanup is now always run from a KUnit-managed test thread.

  - Previously, if a test aborted early, its cleanup (deferred actions,
    resources, and test / suite exit functions) would be run from the main
    KUnit executor thread.

  - This could cause problems if an assertion or crash occurred during cleanup.

  - Now, if the original test thred exits, a new cleanup thread is spawned.

- Improved documentation and examples for test / suite init()/exit() functions.

.. _v6.6:

6.6
===

- Tests and suites can now have 'attributes', which can be filtered upon.

  - 'speed' denotes the expected runtime of a test: tests which are expected to
    take > 1 second to execute can be marked slow with ``KUNIT_CASE_SLOW()``.

  - 'module' lists the name of the module containing the test.

  - See: https://docs.kernel.org/dev-tools/kunit/running_tips.html#test-attributes-and-filtering

- Rust 'doctests' are now automatically converted into and run as KUnit tests.

- Additional architectural features are supported in the arm64 QEMU target.

.. _v6.7:

6.7
===

- Test logs are no longer limited to 2KiB in debugfs. The log will now grow as required.

- KUnit will now warn if a test takes longer than 2 seconds to run, but is not marked as slow.