|
@@ -22,92 +22,92 @@ keep from introducing bugs. The major ones are:
|
|
|
How to run these tests
|
|
|
----------------------
|
|
|
|
|
|
-=== The easy version
|
|
|
+### The easy version
|
|
|
|
|
|
-To run all the tests that come bundled with Tor, run "make check"
|
|
|
+To run all the tests that come bundled with Tor, run `make check`.
|
|
|
|
|
|
To run the Stem tests as well, fetch stem from the git repository,
|
|
|
-set STEM_SOURCE_DIR to the checkout, and run "make test-stem".
|
|
|
+set `STEM_SOURCE_DIR` to the checkout, and run `make test-stem`.
|
|
|
|
|
|
To run the Chutney tests as well, fetch chutney from the git repository,
|
|
|
-set CHUTNEY_PATH to the checkout, and run "make test-network".
|
|
|
+set `CHUTNEY_PATH` to the checkout, and run `make test-network`.
|
|
|
|
|
|
-To run all of the above, run "make test-full".
|
|
|
+To run all of the above, run `make test-full`.
|
|
|
|
|
|
To run all of the above, plus tests that require a working connection to the
|
|
|
-internet, run "make test-full-online".
|
|
|
+internet, run `make test-full-online`.
|
|
|
|
|
|
-=== Running particular subtests
|
|
|
+### Running particular subtests
|
|
|
|
|
|
The Tor unit tests are divided into separate programs and a couple of
|
|
|
bundled unit test programs.
|
|
|
|
|
|
Separate programs are easy. For example, to run the memwipe tests in
|
|
|
-isolation, you just run ./src/test/test-memwipe .
|
|
|
+isolation, you just run `./src/test/test-memwipe`.
|
|
|
|
|
|
To run tests within the unit test programs, you can specify the name
|
|
|
of the test. The string ".." can be used as a wildcard at the end of the
|
|
|
test name. For example, to run all the cell format tests, enter
|
|
|
-"./src/test/test cellfmt/..". To run
|
|
|
+`./src/test/test cellfmt/..`. To run
|
|
|
|
|
|
Many tests that need to mess with global state run in forked subprocesses in
|
|
|
order to keep from contaminating one another. But when debugging a failing test,
|
|
|
you might want to run it without forking a subprocess. To do so, use the
|
|
|
-"--no-fork" option with a single test. (If you specify it along with
|
|
|
+`--no-fork` option with a single test. (If you specify it along with
|
|
|
multiple tests, they might interfere.)
|
|
|
|
|
|
-You can turn on logging in the unit tests by passing one of "--debug",
|
|
|
-"--info", "--notice", or "--warn". By default only errors are displayed.
|
|
|
+You can turn on logging in the unit tests by passing one of `--debug`,
|
|
|
+`--info`, `--notice`, or `--warn`. By default only errors are displayed.
|
|
|
|
|
|
-Unit tests are divided into "./src/test/test" and "./src/test/test-slow".
|
|
|
+Unit tests are divided into `./src/test/test` and `./src/test/test-slow`.
|
|
|
The former are those that should finish in a few seconds; the latter tend to
|
|
|
take more time, and may include CPU-intensive operations, deliberate delays,
|
|
|
and stuff like that.
|
|
|
|
|
|
-=== Finding test coverage
|
|
|
+### Finding test coverage
|
|
|
|
|
|
Test coverage is a measurement of which lines your tests actually visit.
|
|
|
|
|
|
-When you configure Tor with the --enable-coverage option, it should
|
|
|
+When you configure Tor with the `--enable-coverage` option, it should
|
|
|
build with support for coverage in the unit tests, and in a special
|
|
|
-"tor-cov" binary.
|
|
|
+`tor-cov` binary.
|
|
|
|
|
|
Then, run the tests you'd like to see coverage from. If you have old
|
|
|
-coverage output, you may need to run "reset-gcov" first.
|
|
|
+coverage output, you may need to run `reset-gcov` first.
|
|
|
|
|
|
Now you've got a bunch of files scattered around your build directories
|
|
|
-called "*.gcda". In order to extract the coverage output from them, make a
|
|
|
-temporary directory for them and run "./scripts/test/coverage ${TMPDIR}",
|
|
|
-where ${TMPDIR} is the temporary directory you made. This will create a
|
|
|
-".gcov" file for each source file under tests, containing that file's source
|
|
|
+called `*.gcda`. In order to extract the coverage output from them, make a
|
|
|
+temporary directory for them and run `./scripts/test/coverage ${TMPDIR}`,
|
|
|
+where `${TMPDIR}` is the temporary directory you made. This will create a
|
|
|
+`.gcov` file for each source file under tests, containing that file's source
|
|
|
annotated with the number of times the tests hit each line. (You'll need to
|
|
|
have gcov installed.)
|
|
|
|
|
|
You can get a summary of the test coverage for each file by running
|
|
|
-"./scripts/test/cov-display ${TMPDIR}/*" . Each line lists the file's name,
|
|
|
+`./scripts/test/cov-display ${TMPDIR}/*` . Each line lists the file's name,
|
|
|
the number of uncovered lines, the number of uncovered lines, and the
|
|
|
coverage percentage.
|
|
|
|
|
|
For a summary of the test coverage for each _function_, run
|
|
|
-"./scripts/test/cov-display -f ${TMPDIR}/*" .
|
|
|
+`./scripts/test/cov-display -f ${TMPDIR}/*`.
|
|
|
|
|
|
-=== Comparing test coverage
|
|
|
+### Comparing test coverage
|
|
|
|
|
|
Sometimes it's useful to compare test coverage for a branch you're writing to
|
|
|
coverage from another branch (such as git master, for example). But you
|
|
|
-can't run "diff" on the two coverage outputs directly, since the actual
|
|
|
+can't run `diff` on the two coverage outputs directly, since the actual
|
|
|
number of times each line is executed aren't so important, and aren't wholly
|
|
|
deterministic.
|
|
|
|
|
|
Instead, follow the instructions above for each branch, creating a separate
|
|
|
-temporary directory for each. Then, run "./scripts/test/cov-diff ${D1}
|
|
|
-${D2}", where D1 and D2 are the directories you want to compare. This will
|
|
|
+temporary directory for each. Then, run `./scripts/test/cov-diff ${D1}
|
|
|
+${D2}`, where D1 and D2 are the directories you want to compare. This will
|
|
|
produce a diff of the two directories, with all lines normalized to be either
|
|
|
covered or uncovered.
|
|
|
|
|
|
To count new or modified uncovered lines in D2, you can run:
|
|
|
|
|
|
- "./scripts/test/cov-diff ${D1} ${D2}" | grep '^+ *\#' |wc -l
|
|
|
+ ./scripts/test/cov-diff ${D1} ${D2}" | grep '^+ *\#' | wc -l
|
|
|
|
|
|
|
|
|
What kinds of test should I write?
|
|
@@ -132,18 +132,18 @@ Unit and regression tests: Does this function do what it's supposed to?
|
|
|
Most of Tor's unit tests are made using the "tinytest" testing framework.
|
|
|
You can see a guide to using it in the tinytest manual at
|
|
|
|
|
|
- https://github.com/nmathewson/tinytest/blob/master/tinytest-manual.md
|
|
|
+ https://github.com/nmathewson/tinytest/blob/master/tinytest-manual.md
|
|
|
|
|
|
-To add a new test of this kind, either edit an existing C file in src/test/,
|
|
|
+To add a new test of this kind, either edit an existing C file in `src/test/`,
|
|
|
or create a new C file there. Each test is a single function that must
|
|
|
be indexed in the table at the end of the file. We use the label "done:" as
|
|
|
a cleanup point for all test functions.
|
|
|
|
|
|
-(Make sure you read tinytest-manual.md before proceeding.)
|
|
|
+(Make sure you read `tinytest-manual.md` before proceeding.)
|
|
|
|
|
|
I use the term "unit test" and "regression tests" very sloppily here.
|
|
|
|
|
|
-=== A simple example
|
|
|
+### A simple example
|
|
|
|
|
|
Here's an example of a test function for a simple function in util.c:
|
|
|
|
|
@@ -172,20 +172,20 @@ Here's an example of a test function for a simple function in util.c:
|
|
|
|
|
|
This should look pretty familiar to you if you've read the tinytest
|
|
|
manual. One thing to note here is that we use the testing-specific
|
|
|
-function "get_fname" to generate a file with respect to a temporary
|
|
|
+function `get_fname` to generate a file with respect to a temporary
|
|
|
directory that the tests use. You don't need to delete the file;
|
|
|
it will get removed when the tests are done.
|
|
|
|
|
|
-Also note our use of OP_EQ instead of == in the tt_int_op() calls.
|
|
|
-We define OP_* macros to use instead of the binary comparison
|
|
|
+Also note our use of `OP_EQ` instead of `==` in the `tt_int_op()` calls.
|
|
|
+We define `OP_*` macros to use instead of the binary comparison
|
|
|
operators so that analysis tools can more easily parse our code.
|
|
|
-(Coccinelle really hates to see == used as a macro argument.)
|
|
|
+(Coccinelle really hates to see `==` used as a macro argument.)
|
|
|
|
|
|
-Finally, remember that by convention, all *_free() functions that
|
|
|
+Finally, remember that by convention, all `*_free()` functions that
|
|
|
Tor defines are defined to accept NULL harmlessly. Thus, you don't
|
|
|
-need to say "if (contents)" in the cleanup block.
|
|
|
+need to say `if (contents)` in the cleanup block.
|
|
|
|
|
|
-=== Exposing static functions for testing
|
|
|
+### Exposing static functions for testing
|
|
|
|
|
|
Sometimes you need to test a function, but you don't want to expose
|
|
|
it outside its usual module.
|
|
@@ -193,20 +193,20 @@ it outside its usual module.
|
|
|
To support this, Tor's build system compiles a testing version of
|
|
|
each module, with extra identifiers exposed. If you want to
|
|
|
declare a function as static but available for testing, use the
|
|
|
-macro "STATIC" instead of "static". Then, make sure there's a
|
|
|
+macro `STATIC` instead of `static`. Then, make sure there's a
|
|
|
macro-protected declaration of the function in the module's header.
|
|
|
|
|
|
-For example, crypto_curve25519.h contains:
|
|
|
+For example, `crypto_curve25519.h` contains:
|
|
|
|
|
|
-#ifdef CRYPTO_CURVE25519_PRIVATE
|
|
|
-STATIC int curve25519_impl(uint8_t *output, const uint8_t *secret,
|
|
|
+ #ifdef CRYPTO_CURVE25519_PRIVATE
|
|
|
+ STATIC int curve25519_impl(uint8_t *output, const uint8_t *secret,
|
|
|
const uint8_t *basepoint);
|
|
|
-#endif
|
|
|
+ #endif
|
|
|
|
|
|
-The crypto_curve25519.c file and the test_crypto.c file both define
|
|
|
-CRYPTO_CURVE25519_PRIVATE, so they can see this declaration.
|
|
|
+The `crypto_curve25519.c` file and the `test_crypto.c` file both define
|
|
|
+`CRYPTO_CURVE25519_PRIVATE`, so they can see this declaration.
|
|
|
|
|
|
-=== Mock functions for testing in isolation
|
|
|
+### Mock functions for testing in isolation
|
|
|
|
|
|
Often we want to test that a function works right, but the function to
|
|
|
be tested depends on other functions whose behavior is hard to observe,
|
|
@@ -216,7 +216,7 @@ To write tests for this case, you can replace the underlying functions
|
|
|
with testing stubs while your unit test is running. You need to declare
|
|
|
the underlying function as 'mockable', as follows:
|
|
|
|
|
|
- MOCK_DECL(returntype, functionname, (argument list));
|
|
|
+ MOCK_DECL(returntype, functionname, (argument list));
|
|
|
|
|
|
and then later implement it as:
|
|
|
|
|
@@ -229,7 +229,7 @@ For example, if you had a 'connect to remote server' function, you could
|
|
|
declare it as:
|
|
|
|
|
|
|
|
|
- MOCK_DECL(int, connect_to_remote, (const char *name, status_t *status));
|
|
|
+ MOCK_DECL(int, connect_to_remote, (const char *name, status_t *status));
|
|
|
|
|
|
When you declare a function this way, it will be declared as normal in
|
|
|
regular builds, but when the module is built for testing, it is declared
|
|
@@ -238,16 +238,16 @@ as a function pointer initialized to the actual implementation.
|
|
|
In your tests, if you want to override the function with a temporary
|
|
|
replacement, you say:
|
|
|
|
|
|
- MOCK(functionname, replacement_function_name);
|
|
|
+ MOCK(functionname, replacement_function_name);
|
|
|
|
|
|
And later, you can restore the original function with:
|
|
|
|
|
|
- UNMOCK(functionname);
|
|
|
+ UNMOCK(functionname);
|
|
|
|
|
|
For more information, see the definitions of this mocking logic in
|
|
|
-testsupport.h.
|
|
|
+`testsupport.h`.
|
|
|
|
|
|
-=== Okay but what should my tests actually do?
|
|
|
+### Okay but what should my tests actually do?
|
|
|
|
|
|
We talk above about "test coverage" -- making sure that your tests visit
|
|
|
every line of code, or every branch of code. But visiting the code isn't
|
|
@@ -267,11 +267,11 @@ cases and failure csaes.
|
|
|
|
|
|
For example, consider testing this function:
|
|
|
|
|
|
- /** Remove all elements E from sl such that E==element. Preserve
|
|
|
- * the order of any elements before E, but elements after E can be
|
|
|
- * rearranged.
|
|
|
- */
|
|
|
- void smartlist_remove(smartlist_t *sl, const void *element);
|
|
|
+ /** Remove all elements E from sl such that E==element. Preserve
|
|
|
+ * the order of any elements before E, but elements after E can be
|
|
|
+ * rearranged.
|
|
|
+ */
|
|
|
+ void smartlist_remove(smartlist_t *sl, const void *element);
|
|
|
|
|
|
In order to test it well, you should write tests for at least all of the
|
|
|
following cases. (These would be black-box tests, since we're only looking
|
|
@@ -298,19 +298,19 @@ When you consider edge cases, you might try:
|
|
|
|
|
|
Now let's look at the implementation:
|
|
|
|
|
|
- void
|
|
|
- smartlist_remove(smartlist_t *sl, const void *element)
|
|
|
- {
|
|
|
- int i;
|
|
|
- if (element == NULL)
|
|
|
- return;
|
|
|
- for (i=0; i < sl->num_used; i++)
|
|
|
- if (sl->list[i] == element) {
|
|
|
- sl->list[i] = sl->list[--sl->num_used]; /* swap with the end */
|
|
|
- i--; /* so we process the new i'th element */
|
|
|
- sl->list[sl->num_used] = NULL;
|
|
|
- }
|
|
|
- }
|
|
|
+ void
|
|
|
+ smartlist_remove(smartlist_t *sl, const void *element)
|
|
|
+ {
|
|
|
+ int i;
|
|
|
+ if (element == NULL)
|
|
|
+ return;
|
|
|
+ for (i=0; i < sl->num_used; i++)
|
|
|
+ if (sl->list[i] == element) {
|
|
|
+ sl->list[i] = sl->list[--sl->num_used]; /* swap with the end */
|
|
|
+ i--; /* so we process the new i'th element */
|
|
|
+ sl->list[sl->num_used] = NULL;
|
|
|
+ }
|
|
|
+ }
|
|
|
|
|
|
Based on the implementation, we now see three more edge cases to test:
|
|
|
|
|
@@ -319,14 +319,14 @@ Based on the implementation, we now see three more edge cases to test:
|
|
|
* Removing an element from a position other than the end of the list.
|
|
|
|
|
|
|
|
|
-=== What should my tests NOT do?
|
|
|
+### What should my tests NOT do?
|
|
|
|
|
|
Tests shouldn't require a network connection.
|
|
|
|
|
|
Whenever possible, tests shouldn't take more than a second. Put the test
|
|
|
into test/slow if it genuinely needs to be run.
|
|
|
|
|
|
-Tests should not alter global state unless they run with TT_FORK: Tests
|
|
|
+Tests should not alter global state unless they run with `TT_FORK`: Tests
|
|
|
should not require other tests to be run before or after them.
|
|
|
|
|
|
Tests should not leak memory or other resources. To find out if your tests
|
|
@@ -338,16 +338,16 @@ the test should verify that the documented behavior is implemented, but
|
|
|
should not break if other permissible behavior is later implemented.
|
|
|
|
|
|
|
|
|
-=== Advanced techniques: Namespaces
|
|
|
+### Advanced techniques: Namespaces
|
|
|
|
|
|
Sometimes, when you're doing a lot of mocking at once, it's convenient to
|
|
|
isolate your identifiers within a single namespace. If this were C++, we'd
|
|
|
already have namespaces, but for C, we do the best we can with macros and
|
|
|
token-pasting.
|
|
|
|
|
|
-We have some macros defined for this purpose in src/test/test.h. To use
|
|
|
-them, you define NS_MODULE to a prefix to be used for your identifiers, and
|
|
|
-then use other macros in place of identifier names. See src/test/test.h for
|
|
|
+We have some macros defined for this purpose in `src/test/test.h`. To use
|
|
|
+them, you define `NS_MODULE` to a prefix to be used for your identifiers, and
|
|
|
+then use other macros in place of identifier names. See `src/test/test.h` for
|
|
|
more documentation.
|
|
|
|
|
|
|
|
@@ -357,19 +357,19 @@ Integration tests: Calling Tor from the outside
|
|
|
Some tests need to invoke Tor from the outside, and shouldn't run from the
|
|
|
same process as the Tor test program. Reasons for doing this might include:
|
|
|
|
|
|
- * Testing the actual behavior of Tor when run from the command line
|
|
|
- * Testing that a crash-handler correctly logs a stack trace
|
|
|
- * Verifying that violating a sandbox or capability requirement will
|
|
|
- actually crash the program.
|
|
|
- * Needing to run as root in order to test capability inheritance or
|
|
|
- user switching.
|
|
|
+ * Testing the actual behavior of Tor when run from the command line
|
|
|
+ * Testing that a crash-handler correctly logs a stack trace
|
|
|
+ * Verifying that violating a sandbox or capability requirement will
|
|
|
+ actually crash the program.
|
|
|
+ * Needing to run as root in order to test capability inheritance or
|
|
|
+ user switching.
|
|
|
|
|
|
-To add one of these, you generally want a new C program in src/test. Add it
|
|
|
-to TESTS and noinst_PROGRAMS if it can run on its own and return success or
|
|
|
+To add one of these, you generally want a new C program in `src/test`. Add it
|
|
|
+to `TESTS` and `noinst_PROGRAMS` if it can run on its own and return success or
|
|
|
failure. If it needs to be invoked multiple times, or it needs to be
|
|
|
-wrapped, add a new shell script to TESTS, and the new program to
|
|
|
-noinst_PROGRAMS. If you need access to any environment variable from the
|
|
|
-makefile (eg ${PYTHON} for a python interpreter), then make sure that the
|
|
|
+wrapped, add a new shell script to `TESTS`, and the new program to
|
|
|
+`noinst_PROGRAMS`. If you need access to any environment variable from the
|
|
|
+makefile (eg `${PYTHON}` for a python interpreter), then make sure that the
|
|
|
makefile exports them.
|
|
|
|
|
|
Writing integration tests with Stem
|
|
@@ -379,25 +379,25 @@ The 'stem' library includes extensive unit tests for the Tor controller
|
|
|
protocol.
|
|
|
|
|
|
For more information on writing new tests for stem, have a look around
|
|
|
-the test/* directory in stem, and find a good example to emulate. You
|
|
|
+the `test/*` directory in stem, and find a good example to emulate. You
|
|
|
might want to start with
|
|
|
-https://gitweb.torproject.org/stem.git/tree/test/integ/control/controller.py
|
|
|
+`https://gitweb.torproject.org/stem.git/tree/test/integ/control/controller.py`
|
|
|
to improve Tor's test coverage.
|
|
|
|
|
|
-You can run stem tests from tor with "make test-stem", or see
|
|
|
-https://stem.torproject.org/faq.html#how-do-i-run-the-tests .
|
|
|
+You can run stem tests from tor with `make test-stem`, or see
|
|
|
+`https://stem.torproject.org/faq.html#how-do-i-run-the-tests`.
|
|
|
|
|
|
System testing with Chutney
|
|
|
---------------------------
|
|
|
|
|
|
The 'chutney' program configures and launches a set of Tor relays,
|
|
|
-authorities, and clients on your local host. It has a 'test network'
|
|
|
+authorities, and clients on your local host. It has a `test network`
|
|
|
functionality to send traffic through them and verify that the traffic
|
|
|
arrives correctly.
|
|
|
|
|
|
-You can write new test networks by adding them to 'networks'. To add
|
|
|
-them to Tor's tests, add them to the test-network or test-network-all
|
|
|
-targets in Makefile.am.
|
|
|
+You can write new test networks by adding them to `networks`. To add
|
|
|
+them to Tor's tests, add them to the `test-network` or `test-network-all`
|
|
|
+targets in `Makefile.am`.
|
|
|
|
|
|
(Adding new kinds of program to chutney will still require hacking the
|
|
|
code.)
|