| 1 | \input texinfo |
| 2 | @c %**start of header |
| 3 | @setfilename ../../info/ert |
| 4 | @settitle Emacs Lisp Regression Testing |
| 5 | @c %**end of header |
| 6 | |
| 7 | @dircategory Emacs misc features |
| 8 | @direntry |
| 9 | * ERT: (ert). Emacs Lisp regression testing tool. |
| 10 | @end direntry |
| 11 | |
| 12 | @copying |
| 13 | Copyright @copyright{} 2008, 2010--2013 Free Software Foundation, Inc. |
| 14 | |
| 15 | @quotation |
| 16 | Permission is granted to copy, distribute and/or modify this document |
| 17 | under the terms of the GNU Free Documentation License, Version 1.3 or |
| 18 | any later version published by the Free Software Foundation; with no |
| 19 | Invariant Sections, with the Front-Cover texts being ``A GNU Manual,'' |
| 20 | and with the Back-Cover Texts as in (a) below. A copy of the license |
| 21 | is included in the section entitled ``GNU Free Documentation License''. |
| 22 | |
| 23 | (a) The FSF's Back-Cover Text is: ``You have the freedom to copy and |
| 24 | modify this GNU manual.'' |
| 25 | @end quotation |
| 26 | @end copying |
| 27 | |
| 28 | @node Top |
| 29 | @top ERT: Emacs Lisp Regression Testing |
| 30 | |
| 31 | ERT is a tool for automated testing in Emacs Lisp. Its main features |
| 32 | are facilities for defining tests, running them and reporting the |
| 33 | results, and for debugging test failures interactively. |
| 34 | |
| 35 | ERT is similar to tools for other environments such as JUnit, but has |
| 36 | unique features that take advantage of the dynamic and interactive |
| 37 | nature of Emacs. Despite its name, it works well both for test-driven |
| 38 | development (see |
| 39 | @url{http://en.wikipedia.org/wiki/Test-driven_development}) and for |
| 40 | traditional software development methods. |
| 41 | |
| 42 | @menu |
| 43 | * Introduction:: A simple example of an ERT test. |
| 44 | * How to Run Tests:: Run tests in Emacs or from the command line. |
| 45 | * How to Write Tests:: How to add tests to your Emacs Lisp code. |
| 46 | * How to Debug Tests:: What to do if a test fails. |
| 47 | * Extending ERT:: ERT is extensible in several ways. |
| 48 | * Other Testing Concepts:: Features not in ERT. |
| 49 | * GNU Free Documentation License:: The license for this documentation. |
| 50 | |
| 51 | @detailmenu |
| 52 | --- The Detailed Node Listing --- |
| 53 | |
| 54 | How to Run Tests |
| 55 | |
| 56 | * Running Tests Interactively:: Run tests in your current Emacs. |
| 57 | * Running Tests in Batch Mode:: Run tests in emacs -Q. |
| 58 | * Test Selectors:: Choose which tests to run. |
| 59 | |
| 60 | How to Write Tests |
| 61 | |
| 62 | * The @code{should} Macro:: A powerful way to express assertions. |
| 63 | * Expected Failures:: Tests for known bugs. |
| 64 | * Tests and Their Environment:: Don't depend on customizations; no side effects. |
| 65 | * Useful Techniques:: Some examples. |
| 66 | |
| 67 | How to Debug Tests |
| 68 | |
| 69 | * Understanding Explanations:: How ERT gives details on why an assertion failed. |
| 70 | * Interactive Debugging:: Tools available in the ERT results buffer. |
| 71 | |
| 72 | Extending ERT |
| 73 | |
| 74 | * Defining Explanation Functions:: Teach ERT about more predicates. |
| 75 | * Low-Level Functions for Working with Tests:: Use ERT's data for your purposes. |
| 76 | |
| 77 | Other Testing Concepts |
| 78 | |
| 79 | * Mocks and Stubs:: Stubbing out code that is irrelevant to the test. |
| 80 | * Fixtures and Test Suites:: How ERT differs from tools for other languages. |
| 81 | |
| 82 | Appendix |
| 83 | |
| 84 | * GNU Free Documentation License:: The license for this documentation. |
| 85 | |
| 86 | @end detailmenu |
| 87 | @end menu |
| 88 | |
| 89 | @node Introduction |
| 90 | @chapter Introduction |
| 91 | |
| 92 | ERT allows you to define @emph{tests} in addition to functions, |
| 93 | macros, variables, and the other usual Lisp constructs. Tests are |
| 94 | simply Lisp code: code that invokes other code and checks whether |
| 95 | it behaves as expected. |
| 96 | |
| 97 | ERT keeps track of the tests that are defined and provides convenient |
| 98 | commands to run them to verify whether the definitions that are |
| 99 | currently loaded in Emacs pass the tests. |
| 100 | |
| 101 | Some Lisp files have comments like the following (adapted from the |
| 102 | package @code{pp.el}): |
| 103 | |
| 104 | @lisp |
| 105 | ;; (pp-to-string '(quote quote)) ; expected: "'quote" |
| 106 | ;; (pp-to-string '((quote a) (quote b))) ; expected: "('a 'b)\n" |
| 107 | ;; (pp-to-string '('a 'b)) ; same as above |
| 108 | @end lisp |
| 109 | |
| 110 | The code contained in these comments can be evaluated from time to |
| 111 | time to compare the output with the expected output. ERT formalizes |
| 112 | this and introduces a common convention, which simplifies Emacs |
| 113 | development, since programmers no longer have to manually find and |
| 114 | evaluate such comments. |
| 115 | |
| 116 | An ERT test definition equivalent to the above comments is this: |
| 117 | |
| 118 | @lisp |
| 119 | (ert-deftest pp-test-quote () |
| 120 | "Tests the rendering of `quote' symbols in `pp-to-string'." |
| 121 | (should (equal (pp-to-string '(quote quote)) "'quote")) |
| 122 | (should (equal (pp-to-string '((quote a) (quote b))) "('a 'b)\n")) |
| 123 | (should (equal (pp-to-string '('a 'b)) "('a 'b)\n"))) |
| 124 | @end lisp |
| 125 | |
| 126 | If you know @code{defun}, the syntax of @code{ert-deftest} should look |
| 127 | familiar: This example defines a test named @code{pp-test-quote} that |
| 128 | will pass if the three calls to @code{equal} all return true |
| 129 | (non-nil). |
| 130 | |
| 131 | @code{should} is a macro with the same meaning as @code{cl-assert} but |
| 132 | better error reporting. @xref{The @code{should} Macro}. |
| 133 | |
| 134 | Each test should have a name that describes what functionality it tests. |
| 135 | Test names can be chosen arbitrarily---they are in a |
| 136 | namespace separate from functions and variables---but should follow |
| 137 | the usual Emacs Lisp convention of having a prefix that indicates |
| 138 | which package they belong to. Test names are displayed by ERT when |
| 139 | reporting failures and can be used when selecting which tests to run. |
| 140 | |
| 141 | The empty parentheses @code{()} in the first line don't currently have |
| 142 | any meaning and are reserved for future extension. They also make |
| 143 | the syntax of @code{ert-deftest} more similar to that of @code{defun}. |
| 144 | |
| 145 | The docstring describes what feature this test tests. When running |
| 146 | tests interactively, the first line of the docstring is displayed for |
| 147 | tests that fail, so it is good if the first line makes sense on its |
| 148 | own. |
| 149 | |
| 150 | The body of a test can be arbitrary Lisp code. It should have as few |
| 151 | side effects as possible; each test should be written to clean up |
| 152 | after itself, leaving Emacs in the same state as it was before the |
| 153 | test. Tests should clean up even if they fail. @xref{Tests and Their |
| 154 | Environment}. |
| 155 | |
| 156 | |
| 157 | @node How to Run Tests |
| 158 | @chapter How to Run Tests |
| 159 | |
| 160 | You can run tests either in the Emacs you are working in, or on the |
| 161 | command line in a separate Emacs process in batch mode (i.e., with no |
| 162 | user interface). The former mode is convenient during interactive |
| 163 | development, the latter is useful to make sure that tests pass |
| 164 | independently of your customizations; and it allows you to invoke |
| 165 | tests from makefiles, and to write scripts that run tests in several |
| 166 | different Emacs versions. |
| 167 | |
| 168 | @menu |
| 169 | * Running Tests Interactively:: Run tests in your current Emacs. |
| 170 | * Running Tests in Batch Mode:: Run tests in emacs -Q. |
| 171 | * Test Selectors:: Choose which tests to run. |
| 172 | @end menu |
| 173 | |
| 174 | |
| 175 | @node Running Tests Interactively |
| 176 | @section Running Tests Interactively |
| 177 | |
| 178 | You can run the tests that are currently defined in your Emacs with |
| 179 | the command @kbd{@kbd{M-x} ert @kbd{RET} t @kbd{RET}}. (For an |
| 180 | explanation of the @code{t} argument, @pxref{Test Selectors}.) ERT will pop |
| 181 | up a new buffer, the ERT results buffer, showing the results of the |
| 182 | tests run. It looks like this: |
| 183 | |
| 184 | @example |
| 185 | Selector: t |
| 186 | Passed: 31 |
| 187 | Failed: 2 (2 unexpected) |
| 188 | Total: 33/33 |
| 189 | |
| 190 | Started at: 2008-09-11 08:39:25-0700 |
| 191 | Finished. |
| 192 | Finished at: 2008-09-11 08:39:27-0700 |
| 193 | |
| 194 | FF............................... |
| 195 | |
| 196 | F addition-test |
| 197 | (ert-test-failed |
| 198 | ((should |
| 199 | (= |
| 200 | (+ 1 2) |
| 201 | 4)) |
| 202 | :form |
| 203 | (= 3 4) |
| 204 | :value nil)) |
| 205 | |
| 206 | F list-test |
| 207 | (ert-test-failed |
| 208 | ((should |
| 209 | (equal |
| 210 | (list 'a 'b 'c) |
| 211 | '(a b d))) |
| 212 | :form |
| 213 | (equal |
| 214 | (a b c) |
| 215 | (a b d)) |
| 216 | :value nil :explanation |
| 217 | (list-elt 2 |
| 218 | (different-atoms c d)))) |
| 219 | @end example |
| 220 | |
| 221 | At the top, there is a summary of the results: we ran all tests defined |
| 222 | in the current Emacs (@code{Selector: t}), 31 of them passed, and 2 |
| 223 | failed unexpectedly. @xref{Expected Failures}, for an explanation of |
| 224 | the term @emph{unexpected} in this context. |
| 225 | |
| 226 | The line of dots and @code{F}s is a progress bar where each character |
| 227 | represents one test; it fills while the tests are running. A dot |
| 228 | means that the test passed, an @code{F} means that it failed. Below |
| 229 | the progress bar, ERT shows details about each test that had an |
| 230 | unexpected result. In the example above, there are two failures, both |
| 231 | due to failed @code{should} forms. @xref{Understanding Explanations}, |
| 232 | for more details. |
| 233 | |
| 234 | In the ERT results buffer, @kbd{TAB} and @kbd{S-TAB} cycle between |
| 235 | buttons. Each name of a function or macro in this buffer is a button; |
| 236 | moving point to it and typing @kbd{RET} jumps to its definition. |
| 237 | |
| 238 | Pressing @kbd{r} re-runs the test near point on its own. Pressing |
| 239 | @kbd{d} re-runs it with the debugger enabled. @kbd{.} jumps to the |
| 240 | definition of the test near point (@kbd{RET} has the same effect if |
| 241 | point is on the name of the test). On a failed test, @kbd{b} shows |
| 242 | the backtrace of the failure. |
| 243 | |
| 244 | @kbd{l} shows the list of @code{should} forms executed in the test. |
| 245 | If any messages were generated (with the Lisp function @code{message}) |
| 246 | in a test or any of the code that it invoked, @kbd{m} will show them. |
| 247 | |
| 248 | By default, long expressions in the failure details are abbreviated |
| 249 | using @code{print-length} and @code{print-level}. Pressing @kbd{L} |
| 250 | while point is on a test failure will increase the limits to show more |
| 251 | of the expression. |
| 252 | |
| 253 | |
| 254 | @node Running Tests in Batch Mode |
| 255 | @section Running Tests in Batch Mode |
| 256 | |
| 257 | ERT supports automated invocations from the command line or from |
| 258 | scripts or makefiles. There are two functions for this purpose, |
| 259 | @code{ert-run-tests-batch} and @code{ert-run-tests-batch-and-exit}. |
| 260 | They can be used like this: |
| 261 | |
| 262 | @example |
| 263 | emacs -batch -l ert -l my-tests.el -f ert-run-tests-batch-and-exit |
| 264 | @end example |
| 265 | |
| 266 | This command will start up Emacs in batch mode, load ERT, load |
| 267 | @code{my-tests.el}, and run all tests defined in it. It will exit |
| 268 | with a zero exit status if all tests passed, or nonzero if any tests |
| 269 | failed or if anything else went wrong. It will also print progress |
| 270 | messages and error diagnostics to standard output. |
| 271 | |
| 272 | If ERT is not part of your Emacs distribution, you may need to use |
| 273 | @code{-L /path/to/ert/} so that Emacs can find it. You may need |
| 274 | additional @code{-L} flags to ensure that @code{my-tests.el} and all the |
| 275 | files that it requires are on your @code{load-path}. |
| 276 | |
| 277 | |
| 278 | @node Test Selectors |
| 279 | @section Test Selectors |
| 280 | |
| 281 | Functions like @code{ert} accept a @emph{test selector}, a Lisp |
| 282 | expression specifying a set of tests. Test selector syntax is similar |
| 283 | to Common Lisp's type specifier syntax: |
| 284 | |
| 285 | @itemize |
| 286 | @item @code{nil} selects no tests. |
| 287 | @item @code{t} selects all tests. |
| 288 | @item @code{:new} selects all tests that have not been run yet. |
| 289 | @item @code{:failed} and @code{:passed} select tests according to their most recent result. |
| 290 | @item @code{:expected}, @code{:unexpected} select tests according to their most recent result. |
| 291 | @item A string is a regular expression that selects all tests with matching names. |
| 292 | @item A test (i.e., an object of @code{ert-test} data type) selects that test. |
| 293 | @item A symbol selects the test that the symbol names. |
| 294 | @item @code{(member TESTS...)} selects the elements of TESTS, a list of |
| 295 | tests or symbols naming tests. |
| 296 | @item @code{(eql TEST)} selects TEST, a test or a symbol naming a test. |
| 297 | @item @code{(and SELECTORS...)} selects the tests that match all SELECTORS. |
| 298 | @item @code{(or SELECTORS...)} selects the tests that match any SELECTOR. |
| 299 | @item @code{(not SELECTOR)} selects all tests that do not match SELECTOR. |
| 300 | @item @code{(tag TAG)} selects all tests that have TAG on their tags list. |
| 301 | (Tags are optional labels you can apply to tests when you define them.) |
| 302 | @item @code{(satisfies PREDICATE)} selects all tests that satisfy PREDICATE, |
| 303 | a function that takes a test as argument and returns non-nil if it is selected. |
| 304 | @end itemize |
| 305 | |
| 306 | Selectors that are frequently useful when selecting tests to run |
| 307 | include @code{t} to run all tests that are currently defined in Emacs, |
| 308 | @code{"^foo-"} to run all tests in package @code{foo} (this assumes |
| 309 | that package @code{foo} uses the prefix @code{foo-} for its test names), |
| 310 | result-based selectors such as @code{(or :new :unexpected)} to |
| 311 | run all tests that have either not run yet or that had an unexpected |
| 312 | result in the last run, and tag-based selectors such as @code{(not |
| 313 | (tag :causes-redisplay))} to run all tests that are not tagged |
| 314 | @code{:causes-redisplay}. |
| 315 | |
| 316 | |
| 317 | @node How to Write Tests |
| 318 | @chapter How to Write Tests |
| 319 | |
| 320 | ERT lets you define tests in the same way you define functions. You |
| 321 | can type @code{ert-deftest} forms in a buffer and evaluate them there |
| 322 | with @code{eval-defun} or @code{compile-defun}, or you can save the |
| 323 | file and load it, optionally byte-compiling it first. |
| 324 | |
| 325 | Just like @code{find-function} is only able to find where a function |
| 326 | was defined if the function was loaded from a file, ERT is only able |
| 327 | to find where a test was defined if the test was loaded from a file. |
| 328 | |
| 329 | |
| 330 | @menu |
| 331 | * The @code{should} Macro:: A powerful way to express assertions. |
| 332 | * Expected Failures:: Tests for known bugs. |
| 333 | * Tests and Their Environment:: Don't depend on customizations; no side effects. |
| 334 | * Useful Techniques:: Some examples. |
| 335 | @end menu |
| 336 | |
| 337 | @node The @code{should} Macro |
| 338 | @section The @code{should} Macro |
| 339 | |
| 340 | Test bodies can include arbitrary code; but to be useful, they need to |
| 341 | check whether the code being tested (or @emph{code under test}) |
| 342 | does what it is supposed to do. The macro @code{should} is similar to |
| 343 | @code{cl-assert} from the cl package |
| 344 | (@pxref{Assertions,,, cl, Common Lisp Extensions}), |
| 345 | but analyzes its argument form and records information that ERT can |
| 346 | display to help debugging. |
| 347 | |
| 348 | This test definition |
| 349 | |
| 350 | @lisp |
| 351 | (ert-deftest addition-test () |
| 352 | (should (= (+ 1 2) 4))) |
| 353 | @end lisp |
| 354 | |
| 355 | will produce this output when run via @kbd{M-x ert}: |
| 356 | |
| 357 | @example |
| 358 | F addition-test |
| 359 | (ert-test-failed |
| 360 | ((should |
| 361 | (= |
| 362 | (+ 1 2) |
| 363 | 4)) |
| 364 | :form |
| 365 | (= 3 4) |
| 366 | :value nil)) |
| 367 | @end example |
| 368 | |
| 369 | In this example, @code{should} recorded the fact that (= (+ 1 2) 4) |
| 370 | reduced to (= 3 4) before it reduced to nil. When debugging why the |
| 371 | test failed, it helps to know that the function @code{+} returned 3 |
| 372 | here. ERT records the return value for any predicate called directly |
| 373 | within @code{should}. |
| 374 | |
| 375 | In addition to @code{should}, ERT provides @code{should-not}, which |
| 376 | checks that the predicate returns nil, and @code{should-error}, which |
| 377 | checks that the form called within it signals an error. An example |
| 378 | use of @code{should-error}: |
| 379 | |
| 380 | @lisp |
| 381 | (ert-deftest test-divide-by-zero () |
| 382 | (should-error (/ 1 0) |
| 383 | :type 'arith-error)) |
| 384 | @end lisp |
| 385 | |
| 386 | This checks that dividing one by zero signals an error of type |
| 387 | @code{arith-error}. The @code{:type} argument to @code{should-error} |
| 388 | is optional; if absent, any type of error is accepted. |
| 389 | @code{should-error} returns an error description of the error that was |
| 390 | signaled, to allow additional checks to be made. The error |
| 391 | description has the format @code{(ERROR-SYMBOL . DATA)}. |
| 392 | |
| 393 | There is no @code{should-not-error} macro since tests that signal an |
| 394 | error fail anyway, so @code{should-not-error} is effectively the |
| 395 | default. |
| 396 | |
| 397 | @xref{Understanding Explanations}, for more details on what |
| 398 | @code{should} reports. |
| 399 | |
| 400 | |
| 401 | @node Expected Failures |
| 402 | @section Expected Failures |
| 403 | |
| 404 | Some bugs are complicated to fix, or not very important, and are left as |
| 405 | @emph{known bugs}. If there is a test case that triggers the bug and |
| 406 | fails, ERT will alert you of this failure every time you run all |
| 407 | tests. For known bugs, this alert is a distraction. The way to |
| 408 | suppress it is to add @code{:expected-result :failed} to the test |
| 409 | definition: |
| 410 | |
| 411 | @lisp |
| 412 | (ert-deftest future-bug () |
| 413 | "Test `time-forward' with negative arguments. |
| 414 | Since this functionality isn't implemented, the test is known to fail." |
| 415 | :expected-result :failed |
| 416 | (time-forward -1)) |
| 417 | @end lisp |
| 418 | |
| 419 | ERT will still display a small @code{f} in the progress bar as a |
| 420 | reminder that there is a known bug, and will count the test as failed, |
| 421 | but it will be quiet about it otherwise. |
| 422 | |
| 423 | An alternative to marking the test as a known failure this way is to |
| 424 | delete the test. This is a good idea if there is no intent to fix it, |
| 425 | i.e., if the behavior that was formerly considered a bug has become an |
| 426 | accepted feature. |
| 427 | |
| 428 | In general, however, it can be useful to keep tests that are known to |
| 429 | fail. If someone wants to fix the bug, they will have a very good |
| 430 | starting point: an automated test case that reproduces the bug. This |
| 431 | makes it much easier to fix the bug, demonstrate that it is fixed, and |
| 432 | prevent future regressions. |
| 433 | |
| 434 | ERT displays the same kind of alerts for tests that pass unexpectedly |
| 435 | as it displays for unexpected failures. This way, if you make code |
| 436 | changes that happen to fix a bug that you weren't aware of, you will |
| 437 | know to remove the @code{:expected-result} clause of that test and |
| 438 | close the corresponding bug report, if any. |
| 439 | |
| 440 | Since @code{:expected-result} evaluates its argument when the test is |
| 441 | loaded, tests can be marked as known failures only on certain Emacs |
| 442 | versions, specific architectures, etc.: |
| 443 | |
| 444 | @lisp |
| 445 | (ert-deftest foo () |
| 446 | "A test that is expected to fail on Emacs 23 but succeed elsewhere." |
| 447 | :expected-result (if (string-match "GNU Emacs 23[.]" (emacs-version)) |
| 448 | :failed |
| 449 | :passed) |
| 450 | ...) |
| 451 | @end lisp |
| 452 | |
| 453 | |
| 454 | @node Tests and Their Environment |
| 455 | @section Tests and Their Environment |
| 456 | |
| 457 | The outcome of running a test should not depend on the current state |
| 458 | of the environment, and each test should leave its environment in the |
| 459 | same state it found it in. In particular, a test should not depend on |
| 460 | any Emacs customization variables or hooks, and if it has to make any |
| 461 | changes to Emacs's state or state external to Emacs (such as the file |
| 462 | system), it should undo these changes before it returns, regardless of |
| 463 | whether it passed or failed. |
| 464 | |
| 465 | Tests should not depend on the environment because any such |
| 466 | dependencies can make the test brittle or lead to failures that occur |
| 467 | only under certain circumstances and are hard to reproduce. Of |
| 468 | course, the code under test may have settings that affect its |
| 469 | behavior. In that case, it is best to make the test @code{let}-bind |
| 470 | all such setting variables to set up a specific configuration for the |
| 471 | duration of the test. The test can also set up a number of different |
| 472 | configurations and run the code under test with each. |
| 473 | |
| 474 | Tests that have side effects on their environment should restore it to |
| 475 | its original state because any side effects that persist after the |
| 476 | test can disrupt the workflow of the programmer running the tests. If |
| 477 | the code under test has side effects on Emacs's current state, such as |
| 478 | on the current buffer or window configuration, the test should create |
| 479 | a temporary buffer for the code to manipulate (using |
| 480 | @code{with-temp-buffer}), or save and restore the window configuration |
| 481 | (using @code{save-window-excursion}), respectively. For aspects of |
| 482 | the state that can not be preserved with such macros, cleanup should |
| 483 | be performed with @code{unwind-protect}, to ensure that the cleanup |
| 484 | occurs even if the test fails. |
| 485 | |
| 486 | An exception to this are messages that the code under test prints with |
| 487 | @code{message} and similar logging; tests should not bother restoring |
| 488 | the @code{*Message*} buffer to its original state. |
| 489 | |
| 490 | The above guidelines imply that tests should avoid calling highly |
| 491 | customizable commands such as @code{find-file}, except, of course, if |
| 492 | such commands are what they want to test. The exact behavior of |
| 493 | @code{find-file} depends on many settings such as |
| 494 | @code{find-file-wildcards}, @code{enable-local-variables}, and |
| 495 | @code{auto-mode-alist}. It is difficult to write a meaningful test if |
| 496 | its behavior can be affected by so many external factors. Also, |
| 497 | @code{find-file} has side effects that are hard to predict and thus |
| 498 | hard to undo: It may create a new buffer or reuse an existing |
| 499 | buffer if one is already visiting the requested file; and it runs |
| 500 | @code{find-file-hook}, which can have arbitrary side effects. |
| 501 | |
| 502 | Instead, it is better to use lower-level mechanisms with simple and |
| 503 | predictable semantics like @code{with-temp-buffer}, @code{insert} or |
| 504 | @code{insert-file-contents-literally}, and to activate any desired mode |
| 505 | by calling the corresponding function directly, after binding the |
| 506 | hook variables to nil. This avoids the above problems. |
| 507 | |
| 508 | |
| 509 | @node Useful Techniques |
| 510 | @section Useful Techniques when Writing Tests |
| 511 | |
| 512 | Testing simple functions that have no side effects and no dependencies |
| 513 | on their environment is easy. Such tests often look like this: |
| 514 | |
| 515 | @lisp |
| 516 | (ert-deftest ert-test-mismatch () |
| 517 | (should (eql (ert--mismatch "" "") nil)) |
| 518 | (should (eql (ert--mismatch "" "a") 0)) |
| 519 | (should (eql (ert--mismatch "a" "a") nil)) |
| 520 | (should (eql (ert--mismatch "ab" "a") 1)) |
| 521 | (should (eql (ert--mismatch "Aa" "aA") 0)) |
| 522 | (should (eql (ert--mismatch '(a b c) '(a b d)) 2))) |
| 523 | @end lisp |
| 524 | |
| 525 | This test calls the function @code{ert--mismatch} several times with |
| 526 | various combinations of arguments and compares the return value to the |
| 527 | expected return value. (Some programmers prefer @code{(should (eql |
| 528 | EXPECTED ACTUAL))} over the @code{(should (eql ACTUAL EXPECTED))} |
| 529 | shown here. ERT works either way.) |
| 530 | |
| 531 | Here's a more complicated test: |
| 532 | |
| 533 | @lisp |
| 534 | (ert-deftest ert-test-record-backtrace () |
| 535 | (let ((test (make-ert-test :body (lambda () (ert-fail "foo"))))) |
| 536 | (let ((result (ert-run-test test))) |
| 537 | (should (ert-test-failed-p result)) |
| 538 | (with-temp-buffer |
| 539 | (ert--print-backtrace (ert-test-failed-backtrace result)) |
| 540 | (goto-char (point-min)) |
| 541 | (end-of-line) |
| 542 | (let ((first-line (buffer-substring-no-properties |
| 543 | (point-min) (point)))) |
| 544 | (should (equal first-line |
| 545 | " signal(ert-test-failed (\"foo\"))"))))))) |
| 546 | @end lisp |
| 547 | |
| 548 | This test creates a test object using @code{make-ert-test} whose body |
| 549 | will immediately signal failure. It then runs that test and asserts |
| 550 | that it fails. Then, it creates a temporary buffer and invokes |
| 551 | @code{ert--print-backtrace} to print the backtrace of the failed test |
| 552 | to the current buffer. Finally, it extracts the first line from the |
| 553 | buffer and asserts that it matches what we expect. It uses |
| 554 | @code{buffer-substring-no-properties} and @code{equal} to ignore text |
| 555 | properties; for a test that takes properties into account, |
| 556 | @code{buffer-substring} and @code{ert-equal-including-properties} |
| 557 | could be used instead. |
| 558 | |
| 559 | The reason why this test only checks the first line of the backtrace |
| 560 | is that the remainder of the backtrace is dependent on ERT's internals |
| 561 | as well as whether the code is running interpreted or compiled. By |
| 562 | looking only at the first line, the test checks a useful property---that |
| 563 | the backtrace correctly captures the call to @code{signal} that |
| 564 | results from the call to @code{ert-fail}---without being brittle. |
| 565 | |
| 566 | This example also shows that writing tests is much easier if the code |
| 567 | under test was structured with testing in mind. |
| 568 | |
| 569 | For example, if @code{ert-run-test} accepted only symbols that name |
| 570 | tests rather than test objects, the test would need a name for the |
| 571 | failing test, which would have to be a temporary symbol generated with |
| 572 | @code{make-symbol}, to avoid side effects on Emacs's state. Choosing |
| 573 | the right interface for @code{ert-run-tests} allows the test to be |
| 574 | simpler. |
| 575 | |
| 576 | Similarly, if @code{ert--print-backtrace} printed the backtrace to a |
| 577 | buffer with a fixed name rather than the current buffer, it would be |
| 578 | much harder for the test to undo the side effect. Of course, some |
| 579 | code somewhere needs to pick the buffer name. But that logic is |
| 580 | independent of the logic that prints backtraces, and keeping them in |
| 581 | separate functions allows us to test them independently. |
| 582 | |
| 583 | A lot of code that you will encounter in Emacs was not written with |
| 584 | testing in mind. Sometimes, the easiest way to write tests for such |
| 585 | code is to restructure the code slightly to provide better interfaces |
| 586 | for testing. Usually, this makes the interfaces easier to use as |
| 587 | well. |
| 588 | |
| 589 | |
| 590 | @node How to Debug Tests |
| 591 | @chapter How to Debug Tests |
| 592 | |
| 593 | This section describes how to use ERT's features to understand why |
| 594 | a test failed. |
| 595 | |
| 596 | |
| 597 | @menu |
| 598 | * Understanding Explanations:: How ERT gives details on why an assertion failed. |
| 599 | * Interactive Debugging:: Tools available in the ERT results buffer. |
| 600 | @end menu |
| 601 | |
| 602 | |
| 603 | @node Understanding Explanations |
| 604 | @section Understanding Explanations |
| 605 | |
| 606 | Failed @code{should} forms are reported like this: |
| 607 | |
| 608 | @example |
| 609 | F addition-test |
| 610 | (ert-test-failed |
| 611 | ((should |
| 612 | (= |
| 613 | (+ 1 2) |
| 614 | 4)) |
| 615 | :form |
| 616 | (= 3 4) |
| 617 | :value nil)) |
| 618 | @end example |
| 619 | |
| 620 | ERT shows what the @code{should} expression looked like and what |
| 621 | values its subexpressions had: The source code of the assertion was |
| 622 | @code{(should (= (+ 1 2) 4))}, which applied the function @code{=} to |
| 623 | the arguments @code{3} and @code{4}, resulting in the value |
| 624 | @code{nil}. In this case, the test is wrong; it should expect 3 |
| 625 | rather than 4. |
| 626 | |
| 627 | If a predicate like @code{equal} is used with @code{should}, ERT |
| 628 | provides a so-called @emph{explanation}: |
| 629 | |
| 630 | @example |
| 631 | F list-test |
| 632 | (ert-test-failed |
| 633 | ((should |
| 634 | (equal |
| 635 | (list 'a 'b 'c) |
| 636 | '(a b d))) |
| 637 | :form |
| 638 | (equal |
| 639 | (a b c) |
| 640 | (a b d)) |
| 641 | :value nil :explanation |
| 642 | (list-elt 2 |
| 643 | (different-atoms c d)))) |
| 644 | @end example |
| 645 | |
| 646 | In this case, the function @code{equal} was applied to the arguments |
| 647 | @code{(a b c)} and @code{(a b d)}. ERT's explanation shows that |
| 648 | the item at index 2 differs between the two lists; in one list, it is |
| 649 | the atom c, in the other, it is the atom d. |
| 650 | |
| 651 | In simple examples like the above, the explanation is unnecessary. |
| 652 | But in cases where the difference is not immediately apparent, it can |
| 653 | save time: |
| 654 | |
| 655 | @example |
| 656 | F test1 |
| 657 | (ert-test-failed |
| 658 | ((should |
| 659 | (equal x y)) |
| 660 | :form |
| 661 | (equal a a) |
| 662 | :value nil :explanation |
| 663 | (different-symbols-with-the-same-name a a))) |
| 664 | @end example |
| 665 | |
| 666 | ERT only provides explanations for predicates that have an explanation |
| 667 | function registered. @xref{Defining Explanation Functions}. |
| 668 | |
| 669 | |
| 670 | @node Interactive Debugging |
| 671 | @section Interactive Debugging |
| 672 | |
| 673 | Debugging failed tests essentially works the same way as debugging any |
| 674 | other problems with Lisp code. Here are a few tricks specific to |
| 675 | tests: |
| 676 | |
| 677 | @itemize |
| 678 | @item Re-run the failed test a few times to see if it fails in the same way |
| 679 | each time. It's good to find out whether the behavior is |
| 680 | deterministic before spending any time looking for a cause. In the |
| 681 | ERT results buffer, @kbd{r} re-runs the selected test. |
| 682 | |
| 683 | @item Use @kbd{.} to jump to the source code of the test to find out exactly |
| 684 | what it does. Perhaps the test is broken rather than the code |
| 685 | under test. |
| 686 | |
| 687 | @item If the test contains a series of @code{should} forms and you can't |
| 688 | tell which one failed, use @kbd{l}, which shows you the list of all |
| 689 | @code{should} forms executed during the test before it failed. |
| 690 | |
| 691 | @item Use @kbd{b} to view the backtrace. You can also use @kbd{d} to re-run |
| 692 | the test with debugging enabled, this will enter the debugger and show |
| 693 | the backtrace as well; but the top few frames shown there will not be |
| 694 | relevant to you since they are ERT's own debugger hook. @kbd{b} |
| 695 | strips them out, so it is more convenient. |
| 696 | |
| 697 | @item If the test or the code under testing prints messages using |
| 698 | @code{message}, use @kbd{m} to see what messages it printed before it |
| 699 | failed. This can be useful to figure out how far it got. |
| 700 | |
| 701 | @item You can instrument tests for debugging the same way you instrument |
| 702 | @code{defun}s for debugging: go to the source code of the test and |
| 703 | type @kbd{@kbd{C-u} @kbd{C-M-x}}. Then, go back to the ERT buffer and |
| 704 | re-run the test with @kbd{r} or @kbd{d}. |
| 705 | |
| 706 | @item If you have been editing and rearranging tests, it is possible that |
| 707 | ERT remembers an old test that you have since renamed or removed: |
| 708 | renamings or removals of definitions in the source code leave around a |
| 709 | stray definition under the old name in the running process (this is a |
| 710 | common problem in Lisp). In such a situation, hit @kbd{D} to let ERT |
| 711 | forget about the obsolete test. |
| 712 | @end itemize |
| 713 | |
| 714 | |
| 715 | @node Extending ERT |
| 716 | @chapter Extending ERT |
| 717 | |
| 718 | There are several ways to add functionality to ERT. |
| 719 | |
| 720 | @menu |
| 721 | * Defining Explanation Functions:: Teach ERT about more predicates. |
| 722 | * Low-Level Functions for Working with Tests:: Use ERT's data for your purposes. |
| 723 | @end menu |
| 724 | |
| 725 | |
| 726 | @node Defining Explanation Functions |
| 727 | @section Defining Explanation Functions |
| 728 | |
| 729 | The explanation function for a predicate is a function that takes the |
| 730 | same arguments as the predicate and returns an @emph{explanation}. |
| 731 | The explanation should explain why the predicate, when invoked with |
| 732 | the arguments given to the explanation function, returns the value |
| 733 | that it returns. The explanation can be any object but should have a |
| 734 | comprehensible printed representation. If the return value of the |
| 735 | predicate needs no explanation for a given list of arguments, the |
| 736 | explanation function should return nil. |
| 737 | |
| 738 | To associate an explanation function with a predicate, add the |
| 739 | property @code{ert-explainer} to the symbol that names the predicate. |
| 740 | The value of the property should be the symbol that names the |
| 741 | explanation function. |
| 742 | |
| 743 | |
| 744 | @node Low-Level Functions for Working with Tests |
| 745 | @section Low-Level Functions for Working with Tests |
| 746 | |
| 747 | Both @code{ert-run-tests-interactively} and @code{ert-run-tests-batch} |
| 748 | are implemented on top of the lower-level test handling code in the |
| 749 | sections of @file{ert.el} labeled ``Facilities for running a single test'', |
| 750 | ``Test selectors'', and ``Facilities for running a whole set of tests''. |
| 751 | |
| 752 | If you want to write code that works with ERT tests, you should take a |
| 753 | look at this lower-level code. Symbols that start with @code{ert--} |
| 754 | are internal to ERT, whereas those that start with @code{ert-} are |
| 755 | meant to be usable by other code. But there is no mature API yet. |
| 756 | |
| 757 | Contributions to ERT are welcome. |
| 758 | |
| 759 | |
| 760 | @node Other Testing Concepts |
| 761 | @chapter Other Testing Concepts |
| 762 | |
| 763 | For information on mocks, stubs, fixtures, or test suites, see below. |
| 764 | |
| 765 | |
| 766 | @menu |
| 767 | * Mocks and Stubs:: Stubbing out code that is irrelevant to the test. |
| 768 | * Fixtures and Test Suites:: How ERT differs from tools for other languages. |
| 769 | @end menu |
| 770 | |
| 771 | @node Mocks and Stubs |
| 772 | @section Other Tools for Emacs Lisp |
| 773 | |
| 774 | Stubbing out functions or using so-called @emph{mocks} can make it |
| 775 | easier to write tests. See |
| 776 | @url{http://en.wikipedia.org/wiki/Mock_object} for an explanation of |
| 777 | the corresponding concepts in object-oriented languages. |
| 778 | |
| 779 | ERT does not have built-in support for mocks or stubs. The package |
| 780 | @code{el-mock} (see @url{http://www.emacswiki.org/emacs/el-mock.el}) |
| 781 | offers mocks for Emacs Lisp and can be used in conjunction with ERT. |
| 782 | |
| 783 | |
| 784 | @node Fixtures and Test Suites |
| 785 | @section Fixtures and Test Suites |
| 786 | |
| 787 | In many ways, ERT is similar to frameworks for other languages like |
| 788 | SUnit or JUnit. However, two features commonly found in such |
| 789 | frameworks are notably absent from ERT: fixtures and test suites. |
| 790 | |
| 791 | Fixtures are mainly used (e.g., in SUnit or JUnit) to provide an |
| 792 | environment for a set of tests, and consist of set-up and tear-down |
| 793 | functions. |
| 794 | |
| 795 | While fixtures are a useful syntactic simplification in other |
| 796 | languages, this does not apply to Lisp, where higher-order functions |
| 797 | and `unwind-protect' are available. One way to implement and use a |
| 798 | fixture in ERT is |
| 799 | |
| 800 | @lisp |
| 801 | (defun my-fixture (body) |
| 802 | (unwind-protect |
| 803 | (progn [set up] |
| 804 | (funcall body)) |
| 805 | [tear down])) |
| 806 | |
| 807 | (ert-deftest my-test () |
| 808 | (my-fixture |
| 809 | (lambda () |
| 810 | [test code]))) |
| 811 | @end lisp |
| 812 | |
| 813 | (Another way would be a @code{with-my-fixture} macro.) This solves |
| 814 | the set-up and tear-down part, and additionally allows any test |
| 815 | to use any combination of fixtures, so it is more flexible than what |
| 816 | other tools typically allow. |
| 817 | |
| 818 | If the test needs access to the environment the fixture sets up, the |
| 819 | fixture can be modified to pass arguments to the body. |
| 820 | |
| 821 | These are well-known Lisp techniques. Special syntax for them could |
| 822 | be added but would provide only a minor simplification. |
| 823 | |
| 824 | (If you are interested in such syntax, note that splitting set-up and |
| 825 | tear-down into separate functions, like *Unit tools usually do, makes |
| 826 | it impossible to establish dynamic `let' bindings as part of the |
| 827 | fixture. So, blindly imitating the way fixtures are implemented in |
| 828 | other languages would be counter-productive in Lisp.) |
| 829 | |
| 830 | The purpose of test suites is to group related tests together. |
| 831 | |
| 832 | The most common use of this is to run just the tests for one |
| 833 | particular module. Since symbol prefixes are the usual way of |
| 834 | separating module namespaces in Emacs Lisp, test selectors already |
| 835 | solve this by allowing regexp matching on test names; e.g., the |
| 836 | selector "^ert-" selects ERT's self-tests. |
| 837 | |
| 838 | Other uses include grouping tests by their expected execution time, |
| 839 | e.g., to run quick tests during interactive development and slow tests less |
| 840 | often. This can be achieved with the @code{:tag} argument to |
| 841 | @code{ert-deftest} and @code{tag} test selectors. |
| 842 | |
| 843 | @node GNU Free Documentation License |
| 844 | @appendix GNU Free Documentation License |
| 845 | @include doclicense.texi |
| 846 | |
| 847 | @bye |
| 848 | |
| 849 | @c LocalWords: ERT JUnit namespace docstring ERT's |
| 850 | @c LocalWords: backtrace makefiles workflow backtraces API SUnit |
| 851 | @c LocalWords: subexpressions |