src/external/c-tap-harness/README

   1                             C TAP Harness 1.12
   2                (C harness for running TAP-compliant tests)
   3
   4                 Written by Russ Allbery <rra@stanford.edu>
   5
   6   Copyright 2000, 2001, 2004, 2006, 2007, 2008, 2009, 2010, 2011, 2012
   7   Russ Allbery <rra@stanford.edu>.  This software is distributed under a
   8   BSD-style license.  Please see the file LICENSE in the distribution for
   9   more information.
  10
  11 BLURB
  12
  13   C TAP Harness is a pure-C implementation of TAP, the Test Anything
  14   Protocol.  TAP is the text-based protocol used by Perl's test suite.
  15   This package provides a harness similar to Perl's Test::Harness for
  16   running tests, with some additional features useful for test suites in
  17   packages that use Autoconf and Automake, and C and shell libraries to
  18   make writing TAP-compliant test programs easier.
  19
  20 DESCRIPTION
  21
  22   This package started as the runtests program I wrote for INN in 2000 to
  23   serve as the basis for a new test suite using a test protocol similar to
  24   that used for Perl modules.  When I started maintaining additional C
  25   packages, I adopted runtests for the test suite driver of those as well,
  26   resulting in further improvements but also separate copies of the same
  27   program in different distributions.  The C TAP Harness distribution
  28   merges all the various versions into a single code base that all my
  29   packages can pull from.
  30
  31   C TAP Harness provides a full TAP specification (apart from a few
  32   possible edge cases) and has additional special features for supporting
  33   builds outside the source directory.  It's mostly useful for packages
  34   using Autoconf and Automake and because it doesn't assume or require
  35   Perl.
  36
  37   The runtests program can be built with knowledge of the source and build
  38   directory and pass that knowledge on to test scripts, and will search
  39   for test scripts in both the source and build directory.  This makes it
  40   easier for packages using Autoconf and Automake and supporting
  41   out-of-tree builds to build some test programs, ship others, and run
  42   them all regardless of what tree they're in.  It also makes it easier
  43   for test cases to find their supporting files when they run.
  44
  45   Also included in this package are C and shell libraries that provide
  46   utility functions for writing test scripts that use TAP to report
  47   exists.
  48
  49 REQUIREMENTS
  50
  51   C TAP Harness requires a C compiler to build.  Any ISO C89 or later C
  52   compiler on a system supporting the Single UNIX Specification, version 3
  53   (SUSv3) should be sufficient.  This should not be a problem on any
  54   modern system.  The test suite and shell library require a
  55   Bourne-compatible shell.  Outside of the test suite, C TAP Harness has
  56   no other prerequisites or requirements.
  57
  58   To run the test suite, you will need Perl plus the Perl modules
  59   Test::More and Test::Pod.  Test::More comes with Perl 5.8 and later.
  60   Test::Pod is available from CPAN and currently must be installed
  61   separately, but the POD tests will be skipped without interfering with
  62   the rest of the tests if it's not installed.
  63
  64   To check spelling in the POD documentation, Pod::Spell (available from
  65   CPAN) and either aspell or ispell with the american dictionary are also
  66   required.  The user's path is searched for aspell or ispell and aspell
  67   is preferred.  Spelling tests are disabled by default since spelling
  68   dictionaries differ too much between systems.  To enable those tests,
  69   set RRA_MAINTAINER_TESTS to a true value.
  70
  71   To bootstrap from a Git checkout, or if you change the Automake files
  72   and need to regenerate Makefile.in, you will need Automake 1.11 or
  73   later.  For bootstrap or if you change configure.ac or any of the m4
  74   files it includes and need to regenerate configure or config.h.in, you
  75   will need Autoconf 2.64 or later.  Perl is also required to generate the
  76   manual page from a fresh Git checkout.
  77
  78 BUILDING AND TESTING
  79
  80   You can build C TAP Harness and run its internal test suite with:
  81
  82       ./configure
  83       make
  84       make check
  85
  86   While there is a configure script, it exists just to drive the build
  87   system and do some path substitution and isn't doing portability
  88   probes.  Pass --enable-silent-rules to configure for a quieter build
  89   (similar to the Linux kernel).
  90
  91   Use make warnings instead of make to build with full GCC compiler
  92   warnings (requires a relatively current version of GCC).
  93
  94   If a test fails, you can run a single test with verbose output via:
  95
  96       ./runtests -b `pwd`/tests -s `pwd`/tests -o <name-of-test>
  97
  98   Do this instead of running the test program directly since it will
  99   ensure that necessary environment variables are set up.  You may need to
 100   change the -s option if you build with a separate build directory from
 101   the source directory.
 102
 103 USING THE HARNESS
 104
 105   While there is an install target that installs runtests in the default
 106   binary directory (/usr/local/bin by default) and installs the man pages,
 107   one normally wouldn't install anything from this package.  Instead, the
 108   code is intended to be copied into your package and refreshed from the
 109   latest release of C TAP Harness for each release.
 110
 111   You can obviously copy the code and integrate it however works best for
 112   your package and your build system.  Here's how I do it for my packages
 113   as an example:
 114
 115   * Create a tests directory and copy tests/runtests.c into it.  Create a
 116     tests/tap subdirectory and copy the portions of the TAP library (from
 117     tests/tap) that I need for that package into it.  The TAP library is
 118     designed to let you drop in additional source and header files for
 119     additional utility functions that are useful in your package.
 120
 121   * Add code to my top-level Makefile.am (I always use a non-recursive
 122     Makefile with subdir-objects set) to build runtests and the test
 123     library:
 124
 125         check_PROGRAMS = tests/runtests
 126         tests_runtests_CPPFLAGS = -DSOURCE='"$(abs_top_srcdir)/tests"' \
 127                 -DBUILD='"$(abs_top_builddir)/tests"'
 128         check_LIBRARIES = tests/tap/libtap.a
 129         tests_tap_libtap_a_CPPFLAGS = -I$(abs_top_srcdir)/tests
 130         tests_tap_libtap_a_SOURCES = tests/tap/basic.c tests/tap/basic.h \
 131                 tests/tap/float.c tests/tap/float.h tests/tap/macros.h
 132
 133     Omit float.c and float.h from the last line if your package doesn't
 134     need the is_double function.  Building the build and source
 135     directories into runtests will let tests/runtests -o <test> to work
 136     for users without requiring that they set any other variables, even if
 137     they're doing an out-of-source build.
 138
 139     Add additional source files and headers that should go into the TAP
 140     library if you added extra utility functions for your package.
 141
 142   * Add code to Makefile.am to run the test suite:
 143
 144         check-local: $(check_PROGRAMS)
 145               cd tests && ./runtests $(abs_top_srcdir)/tests/TESTS
 146
 147     See the Makefile.am in this package for an example (although note that
 148     it keeps runtests in an unusual location).
 149
 150   * List the test programs in the TESTS file.  This should have the name
 151     of the test executable with the trailing "-t" or ".t" (you can use
 152     either extension as you prefer) omitted.  For any test programs that
 153     need to be compiled, add build rules for them in Makefile.am, simliar
 154     to:
 155
 156         tests_libtap_c_basic_LDADD = tests/tap/libtap.a
 157
 158     and add them to check_PROGRAMS.  If you include the float.c add-on in
 159     your libtap library, you will need to add -lm to the _LDADD setting
 160     for all test programs linked against it.
 161
 162     A more complex example from the remctl package that needs additional
 163     libraries:
 164
 165         tests_client_open_t_LDFLAGS = $(GSSAPI_LDFLAGS)
 166         tests_client_open_t_LDADD = client/libremctl.la tests/tap/libtap.a \
 167                 util/libutil.la $(GSSAPI_LIBS)
 168
 169     If the test program doesn't need to be compiled, add it to EXTRA_DIST
 170     so that it will be included in the distribution.
 171
 172   * If you have test programs written in shell, copy tests/tap/libtap.sh
 173     the tap subdirectory of your tests directory and add it to EXTRA_DIST.
 174     Shell programs should start with:
 175
 176         . "${SOURCE}/tap/libtap.sh"
 177
 178     and can then use the functions defined in the library.
 179
 180   * Optionally copy docs/writing-tests into your package somewhere, such
 181     as tests/README, as instructions to contributors on how to write tests
 182     for this framework.
 183
 184   If you have configuration files that the user must create to enable some
 185   of the tests, conventionally they go into tests/config.
 186
 187   If you have data files that your test cases use, conventionally they go
 188   into tests/data.  You can then find the data directory relative to the
 189   SOURCE environment variable (set by runtests) in your test program.  If
 190   you have data that's compiled or generated by Autoconf, it will be
 191   relative to the BUILD environment variable.  Don't forget to add test
 192   data to EXTRA_DIST as necessary.
 193
 194   For more TAP library add-ons, generally ones that rely on additional
 195   portability code not shipped in this package or with narrower uses, see
 196   the rra-c-util package:
 197
 198       http://www.eyrie.org/~eagle/software/rra-c-util/
 199
 200   There are several additional TAP library add-ons in the tests/tap
 201   directory in that package.  It's also an example of how to use this test
 202   harness in another package.
 203
 204 HOMEPAGE AND SOURCE REPOSITORY
 205
 206   The C TAP Harness web page at:
 207
 208       http://www.eyrie.org/~eagle/software/c-tap-harness/
 209
 210   will always have the current version of this package, the current
 211   documentation, and pointers to any additional resources.
 212
 213   C TAP Harness is maintained using Git.  You can access the current
 214   source by cloning the repository at:
 215
 216       git://git.eyrie.org/devel/c-tap-harness.git
 217
 218   or view the repository via the web at:
 219
 220       http://git.eyrie.org/?p=devel/c-tap-harness.git
 221
 222   C TAP Harness is also available via github at:
 223
 224       http://github.com/rra/c-tap-harness
 225
 226   and the github wiki and issue tracker are available on an experimental
 227   basis.  If you like using the github facilities, try filing issues or
 228   adding supplemental documentation there.