Writing tests for Tor: an incomplete guide
==========================================

Tor uses a variety of testing frameworks and methodologies to try to
keep from introducing bugs.  The major ones are:

   1. Unit tests written in C and shipped with the Tor distribution.

   2. Integration tests written in Python and shipped with the Tor
      distribution.

   3. Integration tests written in Python and shipped with the Stem
      library.  Some of these use the Tor controller protocol.

   4. System tests written in Python and SH, and shipped with the
      Chutney package.  These work by running many instances of Tor
      locally, and sending traffic through them.

   5. The Shadow network simulator.

How to run these tests
----------------------

=== The easy version

To run all the tests that come bundled with Tor, run "make check"

To run the Stem tests as well, fetch stem from the git repository,
set STEM_SOURCE_DIR to the checkout, and run "make test-stem".

To run the Chutney tests as well, fetch chutney from the git repository,
set CHUTNEY_PATH to the checkout, and run "make test-network".

=== Running particular subtests

XXXX WRITEME

=== Finding test coverage

When you configure Tor with the --enable-coverage option, it should
build with support for coverage in the unit tests, and in a special
"tor-cov" binary.  If you launch

XXXX "make test-network" doesn't know about "tor-cov"; you don't get
XXXX coverage from that yet, unless you do "cp src/or/tor-cov
XXXX src/or/tor" before you run it.

What kinds of test should I write?
----------------------------------

XXXX writeme.


Unit and regression tests: Does this function do what it's supposed to?
-----------------------------------------------------------------------

Most of Tor's unit tests are made using the "tinytest" testing framework.
You can see a guide to using it in the tinytest manual at

   https://github.com/nmathewson/tinytest/blob/master/tinytest-manual.md

To add a new test of this kind, either edit an existing C file in src/test/,
or create a new C file there.  Each test is a single function that must
be indexed in the table at the end of the file.  We use the label "done:" as
a cleanup point for all test functions.

(Make sure you read tinytest-manual.md before proceeding.)

I use the term "unit test" and "regression tests" very sloppily here.

=== A simple example

Here's an example of a test function for a simple function in util.c:

    static void
    test_util_writepid(void *arg)
    {
      (void) arg;

      char *contents = NULL;
      const char *fname = get_fname("tmp_pid");
      unsigned long pid;
      char c;

      write_pidfile(fname);

      contents = read_file_to_str(fname, 0, NULL);
      tt_assert(contents);

      int n = sscanf(contents, "%lu\n%c", &pid, &c);
      tt_int_op(n, OP_EQ, 1);
      tt_int_op(pid, OP_EQ, getpid());

    done:
      tor_free(contents);
    }

This should look pretty familiar to you if you've read the tinytest
manual.  One thing to note here is that we use the testing-specific
function "get_fname" to generate a file with respect to a temporary
directory that the tests use.  You don't need to delete the file;
it will get removed when the tests are done.

Also note our use of OP_EQ instead of == in the tt_int_op() calls.
We define OP_* macros to use instead of the binary comparison
operators so that analysis tools can more easily parse our code.
(Coccinelle really hates to see == used as a macro argument.)

Finally, remember that by convention, all *_free() functions that
Tor defines are defined to accept NULL harmlessly.  Thus, you don't
need to say "if (contents)" in the cleanup block.

=== Exposing static functions for testing

Sometimes you need to test a function, but you don't want to expose
it outside its usual module.

To support this, Tor's build system compiles a testing version of
teach module, with extra identifiers exposed.  If you want to
declare a function as static but available for testing, use the
macro "STATIC" instead of "static".  Then, make sure there's a
macro-protected declaration of the function in the module's header.

For example, crypto_curve25519.h contains:

#ifdef CRYPTO_CURVE25519_PRIVATE
STATIC int curve25519_impl(uint8_t *output, const uint8_t *secret,
                           const uint8_t *basepoint);
#endif

The crypto_curve25519.c file and the test_crypto.c file both define
CRYPTO_CURVE25519_PRIVATE, so they can see this declaration.

=== Mock functions for testing in isolation

Often we want to test that a function works right, but the function depends
on other functions whose behavior is hard to observe, or whose 

XXXX WRITEME

=== Advanced techniques: Namespaces


XXXX write this.  danah boyd made us some really awesome stuff here.


Integration tests: Calling Tor from the outside
-----------------------------------------------

XXXX WRITEME

Writing integration tests with Stem
-----------------------------------

XXXX WRITEME

System testing with Chutney
---------------------------

XXXX WRITEME

Who knows what evil lurks in the timings of networks? The Shadow knows!
-----------------------------------------------------------------------

XXXX WRITEME