C TAP Harness 1.12 (C harness for running TAP-compliant tests) Written by Russ Allbery Copyright 2000, 2001, 2004, 2006, 2007, 2008, 2009, 2010, 2011, 2012 Russ Allbery . This software is distributed under a BSD-style license. Please see the file LICENSE in the distribution for more information. BLURB C TAP Harness is a pure-C implementation of TAP, the Test Anything Protocol. TAP is the text-based protocol used by Perl's test suite. This package provides a harness similar to Perl's Test::Harness for running tests, with some additional features useful for test suites in packages that use Autoconf and Automake, and C and shell libraries to make writing TAP-compliant test programs easier. DESCRIPTION This package started as the runtests program I wrote for INN in 2000 to serve as the basis for a new test suite using a test protocol similar to that used for Perl modules. When I started maintaining additional C packages, I adopted runtests for the test suite driver of those as well, resulting in further improvements but also separate copies of the same program in different distributions. The C TAP Harness distribution merges all the various versions into a single code base that all my packages can pull from. C TAP Harness provides a full TAP specification (apart from a few possible edge cases) and has additional special features for supporting builds outside the source directory. It's mostly useful for packages using Autoconf and Automake and because it doesn't assume or require Perl. The runtests program can be built with knowledge of the source and build directory and pass that knowledge on to test scripts, and will search for test scripts in both the source and build directory. This makes it easier for packages using Autoconf and Automake and supporting out-of-tree builds to build some test programs, ship others, and run them all regardless of what tree they're in. It also makes it easier for test cases to find their supporting files when they run. Also included in this package are C and shell libraries that provide utility functions for writing test scripts that use TAP to report exists. REQUIREMENTS C TAP Harness requires a C compiler to build. Any ISO C89 or later C compiler on a system supporting the Single UNIX Specification, version 3 (SUSv3) should be sufficient. This should not be a problem on any modern system. The test suite and shell library require a Bourne-compatible shell. Outside of the test suite, C TAP Harness has no other prerequisites or requirements. To run the test suite, you will need Perl plus the Perl modules Test::More and Test::Pod. Test::More comes with Perl 5.8 and later. Test::Pod is available from CPAN and currently must be installed separately, but the POD tests will be skipped without interfering with the rest of the tests if it's not installed. To check spelling in the POD documentation, Pod::Spell (available from CPAN) and either aspell or ispell with the american dictionary are also required. The user's path is searched for aspell or ispell and aspell is preferred. Spelling tests are disabled by default since spelling dictionaries differ too much between systems. To enable those tests, set RRA_MAINTAINER_TESTS to a true value. To bootstrap from a Git checkout, or if you change the Automake files and need to regenerate Makefile.in, you will need Automake 1.11 or later. For bootstrap or if you change configure.ac or any of the m4 files it includes and need to regenerate configure or config.h.in, you will need Autoconf 2.64 or later. Perl is also required to generate the manual page from a fresh Git checkout. BUILDING AND TESTING You can build C TAP Harness and run its internal test suite with: ./configure make make check While there is a configure script, it exists just to drive the build system and do some path substitution and isn't doing portability probes. Pass --enable-silent-rules to configure for a quieter build (similar to the Linux kernel). Use make warnings instead of make to build with full GCC compiler warnings (requires a relatively current version of GCC). If a test fails, you can run a single test with verbose output via: ./runtests -b `pwd`/tests -s `pwd`/tests -o Do this instead of running the test program directly since it will ensure that necessary environment variables are set up. You may need to change the -s option if you build with a separate build directory from the source directory. USING THE HARNESS While there is an install target that installs runtests in the default binary directory (/usr/local/bin by default) and installs the man pages, one normally wouldn't install anything from this package. Instead, the code is intended to be copied into your package and refreshed from the latest release of C TAP Harness for each release. You can obviously copy the code and integrate it however works best for your package and your build system. Here's how I do it for my packages as an example: * Create a tests directory and copy tests/runtests.c into it. Create a tests/tap subdirectory and copy the portions of the TAP library (from tests/tap) that I need for that package into it. The TAP library is designed to let you drop in additional source and header files for additional utility functions that are useful in your package. * Add code to my top-level Makefile.am (I always use a non-recursive Makefile with subdir-objects set) to build runtests and the test library: check_PROGRAMS = tests/runtests tests_runtests_CPPFLAGS = -DSOURCE='"$(abs_top_srcdir)/tests"' \ -DBUILD='"$(abs_top_builddir)/tests"' check_LIBRARIES = tests/tap/libtap.a tests_tap_libtap_a_CPPFLAGS = -I$(abs_top_srcdir)/tests tests_tap_libtap_a_SOURCES = tests/tap/basic.c tests/tap/basic.h \ tests/tap/float.c tests/tap/float.h tests/tap/macros.h Omit float.c and float.h from the last line if your package doesn't need the is_double function. Building the build and source directories into runtests will let tests/runtests -o to work for users without requiring that they set any other variables, even if they're doing an out-of-source build. Add additional source files and headers that should go into the TAP library if you added extra utility functions for your package. * Add code to Makefile.am to run the test suite: check-local: $(check_PROGRAMS) cd tests && ./runtests $(abs_top_srcdir)/tests/TESTS See the Makefile.am in this package for an example (although note that it keeps runtests in an unusual location). * List the test programs in the TESTS file. This should have the name of the test executable with the trailing "-t" or ".t" (you can use either extension as you prefer) omitted. For any test programs that need to be compiled, add build rules for them in Makefile.am, simliar to: tests_libtap_c_basic_LDADD = tests/tap/libtap.a and add them to check_PROGRAMS. If you include the float.c add-on in your libtap library, you will need to add -lm to the _LDADD setting for all test programs linked against it. A more complex example from the remctl package that needs additional libraries: tests_client_open_t_LDFLAGS = $(GSSAPI_LDFLAGS) tests_client_open_t_LDADD = client/libremctl.la tests/tap/libtap.a \ util/libutil.la $(GSSAPI_LIBS) If the test program doesn't need to be compiled, add it to EXTRA_DIST so that it will be included in the distribution. * If you have test programs written in shell, copy tests/tap/libtap.sh the tap subdirectory of your tests directory and add it to EXTRA_DIST. Shell programs should start with: . "${SOURCE}/tap/libtap.sh" and can then use the functions defined in the library. * Optionally copy docs/writing-tests into your package somewhere, such as tests/README, as instructions to contributors on how to write tests for this framework. If you have configuration files that the user must create to enable some of the tests, conventionally they go into tests/config. If you have data files that your test cases use, conventionally they go into tests/data. You can then find the data directory relative to the SOURCE environment variable (set by runtests) in your test program. If you have data that's compiled or generated by Autoconf, it will be relative to the BUILD environment variable. Don't forget to add test data to EXTRA_DIST as necessary. For more TAP library add-ons, generally ones that rely on additional portability code not shipped in this package or with narrower uses, see the rra-c-util package: http://www.eyrie.org/~eagle/software/rra-c-util/ There are several additional TAP library add-ons in the tests/tap directory in that package. It's also an example of how to use this test harness in another package. HOMEPAGE AND SOURCE REPOSITORY The C TAP Harness web page at: http://www.eyrie.org/~eagle/software/c-tap-harness/ will always have the current version of this package, the current documentation, and pointers to any additional resources. C TAP Harness is maintained using Git. You can access the current source by cloning the repository at: git://git.eyrie.org/devel/c-tap-harness.git or view the repository via the web at: http://git.eyrie.org/?p=devel/c-tap-harness.git C TAP Harness is also available via github at: http://github.com/rra/c-tap-harness and the github wiki and issue tracker are available on an experimental basis. If you like using the github facilities, try filing issues or adding supplemental documentation there.