Commit | Line | Data |
---|---|---|
d221e780 CO |
1 | \input texinfo |
2 | @c %**start of header | |
3 | @setfilename ../../info/ert | |
4 | @settitle Emacs Lisp Regression Testing | |
5 | @c %**end of header | |
6 | ||
7 | @dircategory Emacs | |
8 | @direntry | |
9 | * ERT: (ert). Emacs Lisp Regression Testing. | |
10 | @end direntry | |
11 | ||
12 | @copying | |
73b0cd50 | 13 | Copyright @copyright{} 2008, 2010-2011 Free Software Foundation, Inc. |
d221e780 CO |
14 | |
15 | @quotation | |
16 | Permission is granted to copy, distribute and/or modify this document | |
6872a814 | 17 | under the terms of the GNU Free Documentation License, Version 1.3 or |
d221e780 | 18 | any later version published by the Free Software Foundation; with no |
6872a814 GM |
19 | Invariant Sections, with the Front-Cover texts being ``A GNU Manual,'' |
20 | and with the Back-Cover Texts as in (a) below. A copy of the license | |
21 | is included in the section entitled ``GNU Free Documentation License'' | |
22 | in the Emacs manual. | |
23 | ||
24 | (a) The FSF's Back-Cover Text is: ``You have the freedom to copy and | |
25 | modify this GNU manual. Buying copies from the FSF supports it in | |
26 | developing GNU and promoting software freedom.'' | |
27 | ||
28 | This document is part of a collection distributed under the GNU Free | |
29 | Documentation License. If you want to distribute this document | |
30 | separately from the collection, you can do so by adding a copy of the | |
31 | license to the document, as described in section 6 of the license. | |
d221e780 CO |
32 | @end quotation |
33 | @end copying | |
34 | ||
35 | @node Top, Introduction, (dir), (dir) | |
36 | @top ERT: Emacs Lisp Regression Testing | |
37 | ||
38 | ERT is a tool for automated testing in Emacs Lisp. Its main features | |
39 | are facilities for defining tests, running them and reporting the | |
40 | results, and for debugging test failures interactively. | |
41 | ||
42 | ERT is similar to tools for other environments such as JUnit, but has | |
43 | unique features that take advantage of the dynamic and interactive | |
44 | nature of Emacs. Despite its name, it works well both for test-driven | |
45 | development (see | |
46 | @url{http://en.wikipedia.org/wiki/Test-driven_development}) and for | |
47 | traditional software development methods. | |
48 | ||
49 | @menu | |
50 | * Introduction:: A simple example of an ERT test. | |
51 | * How to Run Tests:: Run tests in your Emacs or from the command line. | |
52 | * How to Write Tests:: How to add tests to your Emacs Lisp code. | |
53 | * How to Debug Tests:: What to do if a test fails. | |
54 | * Extending ERT:: ERT is extensible in several ways. | |
55 | * Other Testing Concepts:: Features not in ERT. | |
56 | ||
57 | @detailmenu | |
58 | --- The Detailed Node Listing --- | |
59 | ||
60 | How to Run Tests | |
61 | ||
62 | * Running Tests Interactively:: Run tests in your current Emacs. | |
63 | * Running Tests in Batch Mode:: Run tests in emacs -Q. | |
64 | * Test Selectors:: Choose which tests to run. | |
65 | ||
66 | How to Write Tests | |
67 | ||
68 | * The @code{should} Macro:: A powerful way to express assertions. | |
69 | * Expected Failures:: Tests for known bugs. | |
70 | * Tests and Their Environment:: Don't depend on customizations; no side effects. | |
71 | * Useful Techniques:: Some examples. | |
72 | ||
73 | How to Debug Tests | |
74 | ||
75 | * Understanding Explanations:: How ERT gives details on why an assertion failed. | |
76 | * Interactive Debugging:: Tools available in the ERT results buffer. | |
77 | ||
78 | Extending ERT | |
79 | ||
80 | * Defining Explanation Functions:: Teach ERT about more predicates. | |
81 | * Low-Level Functions for Working with Tests:: Use ERT's data for your purposes. | |
82 | ||
83 | Other Testing Concepts | |
84 | ||
85 | * Mocks and Stubs:: Stubbing out code that is irrelevant to the test. | |
86 | * Fixtures and Test Suites:: How ERT differs from tools for other languages. | |
87 | ||
88 | @end detailmenu | |
89 | @end menu | |
90 | ||
91 | @node Introduction, How to Run Tests, Top, Top | |
92 | @chapter Introduction | |
93 | ||
94 | ERT allows you to define @emph{tests} in addition to functions, | |
95 | macros, variables, and the other usual Lisp constructs. Tests are | |
96 | simply Lisp code --- code that invokes other code and checks whether | |
97 | it behaves as expected. | |
98 | ||
99 | ERT keeps track of the tests that are defined and provides convenient | |
100 | commands to run them to verify whether the definitions that are | |
101 | currently loaded in Emacs pass the tests. | |
102 | ||
103 | Some Lisp files have comments like the following (adapted from the | |
104 | package @code{pp.el}): | |
105 | ||
106 | @lisp | |
107 | ;; (pp-to-string '(quote quote)) ; expected: "'quote" | |
108 | ;; (pp-to-string '((quote a) (quote b))) ; expected: "('a 'b)\n" | |
109 | ;; (pp-to-string '('a 'b)) ; same as above | |
110 | @end lisp | |
111 | ||
112 | The code contained in these comments can be evaluated from time to | |
113 | time to compare the output with the expected output. ERT formalizes | |
114 | this and introduces a common convention, which simplifies Emacs | |
115 | development, since programmers no longer have to manually find and | |
116 | evaluate such comments. | |
117 | ||
118 | An ERT test definition equivalent to the above comments is this: | |
119 | ||
120 | @lisp | |
121 | (ert-deftest pp-test-quote () | |
122 | "Tests the rendering of `quote' symbols in `pp-to-string'." | |
123 | (should (equal (pp-to-string '(quote quote)) "'quote")) | |
124 | (should (equal (pp-to-string '((quote a) (quote b))) "('a 'b)\n")) | |
125 | (should (equal (pp-to-string '('a 'b)) "('a 'b)\n"))) | |
126 | @end lisp | |
127 | ||
128 | If you know @code{defun}, the syntax of @code{ert-deftest} should look | |
129 | familiar: This example defines a test named @code{pp-test-quote} that | |
130 | will pass if the three calls to @code{equal} all return true | |
131 | (non-nil). | |
132 | ||
133 | @code{should} is a macro with the same meaning as @code{assert} but | |
134 | better error reporting. @xref{The @code{should} Macro}. | |
135 | ||
136 | Each test should have a name that describes what functionality the | |
137 | test tests. Test names can be chosen arbitrarily --- they are in a | |
138 | namespace separate from functions and variables --- but should follow | |
139 | the usual Emacs Lisp convention of having a prefix that indicates | |
140 | which package they belong to. Test names are displayed by ERT when | |
141 | reporting failures and can be used when selecting which tests to run. | |
142 | ||
143 | The empty parentheses @code{()} in the first line don't currently have | |
144 | any meaning and are reserved for future extension. They also make | |
145 | @code{ert-deftest}'s syntax more similar to @code{defun}. | |
146 | ||
147 | The docstring describes what feature this test tests. When running | |
148 | tests interactively, the first line of the docstring is displayed for | |
149 | tests that fail, so it is good if the first line makes sense on its | |
150 | own. | |
151 | ||
152 | The body of a test can be arbitrary Lisp code. It should have as few | |
153 | side effects as possible; each test should be written to clean up | |
154 | after itself, leaving Emacs in the same state as it was before the | |
155 | test. Tests should clean up even if they fail. @xref{Tests and Their | |
156 | Environment}. | |
157 | ||
158 | ||
159 | @node How to Run Tests, How to Write Tests, Introduction, Top | |
160 | @chapter How to Run Tests | |
161 | ||
162 | You can run tests either in the Emacs you are working in, or on the | |
163 | command line in a separate Emacs process in batch mode (i.e., with no | |
164 | user interface). The former mode is convenient during interactive | |
165 | development, the latter is useful to make sure that tests pass | |
166 | independently of your customizations, allows tests to be invoked from | |
167 | makefiles and scripts to be written that run tests in several | |
168 | different Emacs versions. | |
169 | ||
170 | @menu | |
171 | * Running Tests Interactively:: Run tests in your current Emacs. | |
172 | * Running Tests in Batch Mode:: Run tests in emacs -Q. | |
173 | * Test Selectors:: Choose which tests to run. | |
174 | @end menu | |
175 | ||
176 | ||
177 | @node Running Tests Interactively, Running Tests in Batch Mode, How to Run Tests, How to Run Tests | |
178 | @section Running Tests Interactively | |
179 | ||
180 | You can run the tests that are currently defined in your Emacs with | |
181 | the command @kbd{@kbd{M-x} ert @kbd{RET} t @kbd{RET}}. ERT will pop | |
182 | up a new buffer, the ERT results buffer, showing the results of the | |
183 | tests run. It looks like this: | |
184 | ||
185 | @example | |
186 | Selector: t | |
187 | Passed: 31 | |
188 | Failed: 2 (2 unexpected) | |
189 | Total: 33/33 | |
190 | ||
191 | Started at: 2008-09-11 08:39:25-0700 | |
192 | Finished. | |
193 | Finished at: 2008-09-11 08:39:27-0700 | |
194 | ||
195 | FF............................... | |
196 | ||
197 | F addition-test | |
198 | (ert-test-failed | |
199 | ((should | |
200 | (= | |
201 | (+ 1 2) | |
202 | 4)) | |
203 | :form | |
204 | (= 3 4) | |
205 | :value nil)) | |
206 | ||
207 | F list-test | |
208 | (ert-test-failed | |
209 | ((should | |
210 | (equal | |
211 | (list 'a 'b 'c) | |
212 | '(a b d))) | |
213 | :form | |
214 | (equal | |
215 | (a b c) | |
216 | (a b d)) | |
217 | :value nil :explanation | |
218 | (list-elt 2 | |
219 | (different-atoms c d)))) | |
220 | @end example | |
221 | ||
222 | At the top, there is a summary of the results: We ran all tests in the | |
223 | current Emacs (@code{Selector: t}), 31 of them passed, and 2 failed | |
224 | unexpectedly. @xref{Expected Failures}, for an explanation of the | |
225 | term @emph{unexpected} in this context. | |
226 | ||
227 | The line of dots and @code{F}s is a progress bar where each character | |
228 | represents one test; it fills while the tests are running. A dot | |
229 | means that the test passed, an @code{F} means that it failed. Below | |
230 | the progress bar, ERT shows details about each test that had an | |
231 | unexpected result. In the example above, there are two failures, both | |
232 | due to failed @code{should} forms. @xref{Understanding Explanations}, | |
233 | for more details. | |
234 | ||
235 | In the ERT results buffer, @kbd{TAB} and @kbd{S-TAB} cycle between | |
236 | buttons. Each name of a function or macro in this buffer is a button; | |
237 | moving point to it and typing @kbd{RET} jumps to its definition. | |
238 | ||
239 | Pressing @kbd{r} re-runs the test near point on its own. Pressing | |
240 | @kbd{d} re-runs it with the debugger enabled. @kbd{.} jumps to the | |
241 | definition of the test near point (@kbd{RET} has the same effect if | |
242 | point is on the name of the test). On a failed test, @kbd{b} shows | |
243 | the backtrace of the failure. | |
244 | ||
245 | @kbd{l} shows the list of @code{should} forms executed in the test. | |
246 | If any messages were generated (with the Lisp function @code{message}) | |
247 | in a test or any of the code that it invoked, @kbd{m} will show them. | |
248 | ||
249 | By default, long expressions in the failure details are abbreviated | |
250 | using @code{print-length} and @code{print-level}. Pressing @kbd{L} | |
251 | while point is on a test failure will increase the limits to show more | |
252 | of the expression. | |
253 | ||
254 | ||
255 | @node Running Tests in Batch Mode, Test Selectors, Running Tests Interactively, How to Run Tests | |
256 | @section Running Tests in Batch Mode | |
257 | ||
258 | ERT supports automated invocations from the command line or from | |
259 | scripts or makefiles. There are two functions for this purpose, | |
260 | @code{ert-run-tests-batch} and @code{ert-run-tests-batch-and-exit}. | |
261 | They can be used like this: | |
262 | ||
263 | @example | |
264 | emacs -batch -L /path/to/ert -l ert.el -l my-tests.el -f ert-run-tests-batch-and-exit | |
265 | @end example | |
266 | ||
267 | This command will start up Emacs in batch mode, load ERT, load | |
268 | @code{my-tests.el}, and run all tests defined in it. It will exit | |
269 | with a zero exit status if all tests passed, or nonzero if any tests | |
270 | failed or if anything else went wrong. It will also print progress | |
271 | messages and error diagnostics to standard output. | |
272 | ||
273 | You may need additional @code{-L} flags to ensure that | |
274 | @code{my-tests.el} and all the files that it requires are on your | |
275 | @code{load-path}. | |
276 | ||
277 | ||
278 | @node Test Selectors, , Running Tests in Batch Mode, How to Run Tests | |
279 | @section Test Selectors | |
280 | ||
281 | Functions like @code{ert} accept a @emph{test selector}, a Lisp | |
282 | expression specifying a set of tests. Test selector syntax is similar | |
283 | to Common Lisp's type specifier syntax: | |
284 | ||
285 | @itemize | |
286 | @item @code{nil} selects no tests. | |
287 | @item @code{t} selects all tests. | |
288 | @item @code{:new} selects all tests that have not been run yet. | |
289 | @item @code{:failed} and @code{:passed} select tests according to their most recent result. | |
290 | @item @code{:expected}, @code{:unexpected} select tests according to their most recent result. | |
291 | @item A string selects all tests that have a name that matches the string, a regexp. | |
292 | @item A test selects that test. | |
293 | @item A symbol selects the test that the symbol names. | |
294 | @item @code{(member TESTS...)} selects TESTS, a list of tests or symbols naming tests. | |
295 | @item @code{(eql TEST)} selects TEST, a test or a symbol naming a test. | |
296 | @item @code{(and SELECTORS...)} selects the tests that match all SELECTORS. | |
297 | @item @code{(or SELECTORS...)} selects the tests that match any SELECTOR. | |
298 | @item @code{(not SELECTOR)} selects all tests that do not match SELECTOR. | |
299 | @item @code{(tag TAG)} selects all tests that have TAG on their tags list. | |
300 | @item @code{(satisfies PREDICATE)} Selects all tests that satisfy PREDICATE. | |
301 | @end itemize | |
302 | ||
303 | Selectors that are frequently useful when selecting tests to run | |
304 | include @code{t} to run all tests that are currently defined in Emacs, | |
305 | @code{"^foo-"} to run all tests in package @code{foo} --- this assumes | |
306 | that package @code{foo} uses the prefix @code{foo-} for its test names | |
307 | ---, result-based selectors such as @code{(or :new :unexpected)} to | |
308 | run all tests that have either not run yet or that had an unexpected | |
309 | result in the last run, and tag-based selectors such as @code{(not | |
310 | (tag :causes-redisplay))} to run all tests that are not tagged | |
311 | @code{:causes-redisplay}. | |
312 | ||
313 | ||
314 | @node How to Write Tests, How to Debug Tests, How to Run Tests, Top | |
315 | @chapter How to Write Tests | |
316 | ||
317 | ERT lets you define tests in the same way you define functions. You | |
318 | can type @code{ert-deftest} forms in a buffer and evaluate them there | |
319 | with @code{eval-defun} or @code{compile-defun}, or you can save the | |
320 | file and load it, optionally byte-compiling it first. | |
321 | ||
322 | Just like @code{find-function} is only able to find where a function | |
323 | was defined if the function was loaded from a file, ERT is only able | |
324 | to find where a test was defined if the test was loaded from a file. | |
325 | ||
326 | ||
327 | @menu | |
328 | * The @code{should} Macro:: A powerful way to express assertions. | |
329 | * Expected Failures:: Tests for known bugs. | |
330 | * Tests and Their Environment:: Don't depend on customizations; no side effects. | |
331 | * Useful Techniques:: Some examples. | |
332 | @end menu | |
333 | ||
334 | @node The @code{should} Macro, Expected Failures, How to Write Tests, How to Write Tests | |
335 | @section The @code{should} Macro | |
336 | ||
337 | Test bodies can include arbitrary code; but to be useful, they need to | |
338 | have checks whether the code being tested (or @emph{code under test}) | |
339 | does what it is supposed to do. The macro @code{should} is similar to | |
340 | @code{assert} from the cl package, but analyzes its argument form and | |
341 | records information that ERT can display to help debugging. | |
342 | ||
343 | This test definition | |
344 | ||
345 | @lisp | |
346 | (ert-deftest addition-test () | |
347 | (should (= (+ 1 2) 4))) | |
348 | @end lisp | |
349 | ||
350 | will produce this output when run via @kbd{M-x ert}: | |
351 | ||
352 | @example | |
353 | F addition-test | |
354 | (ert-test-failed | |
355 | ((should | |
356 | (= | |
357 | (+ 1 2) | |
358 | 4)) | |
359 | :form | |
360 | (= 3 4) | |
361 | :value nil)) | |
362 | @end example | |
363 | ||
364 | In this example, @code{should} recorded the fact that (= (+ 1 2) 4) | |
365 | reduced to (= 3 4) before it reduced to nil. When debugging why the | |
366 | test failed, it helps to know that the function @code{+} returned 3 | |
367 | here. ERT records the return value for any predicate called directly | |
368 | within @code{should}. | |
369 | ||
370 | In addition to @code{should}, ERT provides @code{should-not}, which | |
371 | checks that the predicate returns nil, and @code{should-error}, which | |
372 | checks that the form called within it signals an error. An example | |
373 | use of @code{should-error}: | |
374 | ||
375 | @lisp | |
376 | (ert-deftest test-divide-by-zero () | |
377 | (should-error (/ 1 0) | |
378 | :type 'arith-error)) | |
379 | @end lisp | |
380 | ||
381 | This checks that dividing one by zero signals an error of type | |
382 | @code{arith-error}. The @code{:type} argument to @code{should-error} | |
383 | is optional; if absent, any type of error is accepted. | |
384 | @code{should-error} returns an error description of the error that was | |
385 | signalled, to allow additional checks to be made. The error | |
386 | description has the format @code{(ERROR-SYMBOL . DATA)}. | |
387 | ||
388 | There is no @code{should-not-error} macro since tests that signal an | |
389 | error fail anyway, so @code{should-not-error} is effectively the | |
390 | default. | |
391 | ||
392 | @xref{Understanding Explanations}, for more details on what | |
393 | @code{should} reports. | |
394 | ||
395 | ||
396 | @node Expected Failures, Tests and Their Environment, The @code{should} Macro, How to Write Tests | |
397 | @section Expected Failures | |
398 | ||
399 | Some bugs are complicated to fix or not very important and are left as | |
400 | @emph{known bugs}. If there is a test case that triggers the bug and | |
401 | fails, ERT will alert you of this failure every time you run all | |
402 | tests. For known bugs, this alert is a distraction. The way to | |
403 | suppress it is to add @code{:expected-result :failed} to the test | |
404 | definition: | |
405 | ||
406 | @lisp | |
407 | (ert-deftest future-bug () | |
408 | "Test `time-forward' with negative arguments. | |
409 | Since this functionality isn't implemented yet, the test is known to fail." | |
410 | :expected-result :failed | |
411 | (time-forward -1)) | |
412 | @end lisp | |
413 | ||
414 | ERT will still display a small @code{f} in the progress bar as a | |
415 | reminder that there is a known bug, and will count the test as failed, | |
416 | but it will be quiet about it otherwise. | |
417 | ||
418 | An alternative to marking the test as a known failure this way is to | |
419 | delete the test. This is a good idea if there is no intent to fix it, | |
420 | i.e., if the behavior that was formerly considered a bug has become an | |
421 | accepted feature. | |
422 | ||
423 | In general, however, it can be useful to keep tests that are known to | |
424 | fail. If someone wants to fix the bug, they will have a very good | |
425 | starting point: an automated test case that reproduces the bug. This | |
426 | makes it much easier to fix the bug, demonstrate that it is fixed, and | |
427 | prevent future regressions. | |
428 | ||
429 | ERT displays the same kind of alerts for tests that pass unexpectedly | |
430 | that it displays for unexpected failures. This way, if you make code | |
431 | changes that happen to fix a bug that you weren't aware of, you will | |
432 | know to remove the @code{:expected-result} clause of that test and | |
433 | close the corresponding bug report, if any. | |
434 | ||
435 | Since @code{:expected-result} evaluates its argument when the test is | |
436 | loaded, tests can be marked as known failures only on certain Emacs | |
437 | versions, specific architectures, etc.: | |
438 | ||
439 | @lisp | |
440 | (ert-deftest foo () | |
441 | "A test that is expected to fail on Emacs 23 but succeed elsewhere." | |
442 | :expected-result (if (string-match "GNU Emacs 23[.]" (emacs-version)) | |
443 | :failed | |
444 | :passed) | |
445 | ...) | |
446 | @end lisp | |
447 | ||
448 | ||
449 | @node Tests and Their Environment, Useful Techniques, Expected Failures, How to Write Tests | |
450 | @section Tests and Their Environment | |
451 | ||
452 | The outcome of running a test should not depend on the current state | |
453 | of the environment, and each test should leave its environment in the | |
454 | same state it found it in. In particular, a test should not depend on | |
455 | any Emacs customization variables or hooks, and if it has to make any | |
456 | changes to Emacs' state or state external to Emacs such as the file | |
457 | system, it should undo these changes before it returns, regardless of | |
458 | whether it passed or failed. | |
459 | ||
460 | Tests should not depend on the environment because any such | |
461 | dependencies can make the test brittle or lead to failures that occur | |
462 | only under certain circumstances and are hard to reproduce. Of | |
463 | course, the code under test may have settings that affect its | |
464 | behavior. In that case, it is best to make the test @code{let}-bind | |
465 | all such settings variables to set up a specific configuration for the | |
466 | duration of the test. The test can also set up a number of different | |
467 | configurations and run the code under test with each. | |
468 | ||
469 | Tests that have side effects on their environment should restore it to | |
470 | its original state because any side effects that persist after the | |
471 | test can disrupt the workflow of the programmer running the tests. If | |
472 | the code under test has side effects on Emacs' current state, such as | |
473 | on the current buffer or window configuration, the test should create | |
474 | a temporary buffer for the code to manipulate (using | |
475 | @code{with-temp-buffer}), or save and restore the window configuration | |
476 | (using @code{save-window-excursion}), respectively. For aspects of | |
477 | the state that can not be preserved with such macros, cleanup should | |
478 | be performed with @code{unwind-protect}, to ensure that the cleanup | |
479 | occurs even if the test fails. | |
480 | ||
481 | An exception to this are messages that the code under test prints with | |
482 | @code{message} and similar logging; tests should not bother restoring | |
483 | the @code{*Message*} buffer to its original state. | |
484 | ||
485 | The above guidelines imply that tests should avoid calling highly | |
486 | customizable commands such as @code{find-file}, except, of course, if | |
487 | such commands are what they want to test. The exact behavior of | |
488 | @code{find-file} depends on many settings such as | |
489 | @code{find-file-wildcards}, @code{enable-local-variables}, and | |
490 | @code{auto-mode-alist}. It is difficult to write a meaningful test if | |
491 | its behavior can be affected by so many external factors. Also, | |
492 | @code{find-file} has side effects that are hard to predict and thus | |
493 | hard to undo: It may create a new buffer or may reuse an existing | |
494 | buffer if one is already visiting the requested file; and it runs | |
495 | @code{find-file-hook}, which can have arbitrary side effects. | |
496 | ||
497 | Instead, it is better to use lower-level mechanisms with simple and | |
498 | predictable semantics like @code{with-temp-buffer}, @code{insert} or | |
499 | @code{insert-file-contents-literally}, and activating the desired mode | |
500 | by calling the corresponding function directly --- after binding the | |
501 | hook variables to nil. This avoids the above problems. | |
502 | ||
503 | ||
504 | @node Useful Techniques, , Tests and Their Environment, How to Write Tests | |
505 | @section Useful Techniques when Writing Tests | |
506 | ||
507 | Testing simple functions that have no side effects and no dependencies | |
508 | on their environment is easy. Such tests often look like this: | |
509 | ||
510 | @lisp | |
511 | (ert-deftest ert-test-mismatch () | |
512 | (should (eql (ert--mismatch "" "") nil)) | |
513 | (should (eql (ert--mismatch "" "a") 0)) | |
514 | (should (eql (ert--mismatch "a" "a") nil)) | |
515 | (should (eql (ert--mismatch "ab" "a") 1)) | |
516 | (should (eql (ert--mismatch "Aa" "aA") 0)) | |
517 | (should (eql (ert--mismatch '(a b c) '(a b d)) 2))) | |
518 | @end lisp | |
519 | ||
520 | This test calls the function @code{ert--mismatch} several times with | |
521 | various combinations of arguments and compares the return value to the | |
522 | expected return value. (Some programmers prefer @code{(should (eql | |
523 | EXPECTED ACTUAL))} over the @code{(should (eql ACTUAL EXPECTED))} | |
524 | shown here. ERT works either way.) | |
525 | ||
526 | Here's a more complicated test: | |
527 | ||
528 | @lisp | |
529 | (ert-deftest ert-test-record-backtrace () | |
530 | (let ((test (make-ert-test :body (lambda () (ert-fail "foo"))))) | |
531 | (let ((result (ert-run-test test))) | |
532 | (should (ert-test-failed-p result)) | |
533 | (with-temp-buffer | |
534 | (ert--print-backtrace (ert-test-failed-backtrace result)) | |
535 | (goto-char (point-min)) | |
536 | (end-of-line) | |
537 | (let ((first-line (buffer-substring-no-properties (point-min) (point)))) | |
538 | (should (equal first-line " signal(ert-test-failed (\"foo\"))"))))))) | |
539 | @end lisp | |
540 | ||
541 | This test creates a test object using @code{make-ert-test} whose body | |
542 | will immediately signal failure. It then runs that test and asserts | |
543 | that it fails. Then, it creates a temporary buffer and invokes | |
544 | @code{ert--print-backtrace} to print the backtrace of the failed test | |
545 | to the current buffer. Finally, it extracts the first line from the | |
546 | buffer and asserts that it matches what we expect. It uses | |
547 | @code{buffer-substring-no-properties} and @code{equal} to ignore text | |
548 | properties; for a test that takes properties into account, | |
549 | @code{buffer-substring} and @code{ert-equal-including-properties} | |
550 | could be used instead. | |
551 | ||
552 | The reason why this test only checks the first line of the backtrace | |
553 | is that the remainder of the backtrace is dependent on ERT's internals | |
554 | as well as whether the code is running interpreted or compiled. By | |
555 | looking only at the first line, the test checks a useful property | |
556 | --- that the backtrace correctly captures the call to @code{signal} that | |
557 | results from the call to @code{ert-fail} --- without being brittle. | |
558 | ||
559 | This example also shows that writing tests is much easier if the code | |
560 | under test was structured with testing in mind. | |
561 | ||
562 | For example, if @code{ert-run-test} accepted only symbols that name | |
563 | tests rather than test objects, the test would need a name for the | |
564 | failing test, which would have to be a temporary symbol generated with | |
565 | @code{make-symbol}, to avoid side effects on Emacs' state. Choosing | |
566 | the right interface for @code{ert-run-tests} allows the test to be | |
567 | simpler. | |
568 | ||
569 | Similarly, if @code{ert--print-backtrace} printed the backtrace to a | |
570 | buffer with a fixed name rather than the current buffer, it would be | |
571 | much harder for the test to undo the side effect. Of course, some | |
572 | code somewhere needs to pick the buffer name. But that logic is | |
573 | independent of the logic that prints backtraces, and keeping them in | |
574 | separate functions allows us to test them independently. | |
575 | ||
576 | A lot of code that you will encounter in Emacs was not written with | |
577 | testing in mind. Sometimes, the easiest way to write tests for such | |
578 | code is to restructure the code slightly to provide better interfaces | |
579 | for testing. Usually, this makes the interfaces easier to use as | |
580 | well. | |
581 | ||
582 | ||
583 | @node How to Debug Tests, Extending ERT, How to Write Tests, Top | |
584 | @chapter How to Debug Tests | |
585 | ||
586 | This section describes how to use ERT's features to understand why | |
587 | a test failed. | |
588 | ||
589 | ||
590 | @menu | |
591 | * Understanding Explanations:: How ERT gives details on why an assertion failed. | |
592 | * Interactive Debugging:: Tools available in the ERT results buffer. | |
593 | @end menu | |
594 | ||
595 | ||
596 | @node Understanding Explanations, Interactive Debugging, How to Debug Tests, How to Debug Tests | |
597 | @section Understanding Explanations | |
598 | ||
599 | Failed @code{should} forms are reported like this: | |
600 | ||
601 | @example | |
602 | F addition-test | |
603 | (ert-test-failed | |
604 | ((should | |
605 | (= | |
606 | (+ 1 2) | |
607 | 4)) | |
608 | :form | |
609 | (= 3 4) | |
610 | :value nil)) | |
611 | @end example | |
612 | ||
613 | ERT shows what the @code{should} expression looked like and what | |
614 | values its subexpressions had: The source code of the assertion was | |
615 | @code{(should (= (+ 1 2) 4))}, which applied the function @code{=} to | |
616 | the arguments @code{3} and @code{4}, resulting in the value | |
617 | @code{nil}. In this case, the test is wrong; it should expect 3 | |
618 | rather than 4. | |
619 | ||
620 | If a predicate like @code{equal} is used with @code{should}, ERT | |
621 | provides a so-called @emph{explanation}: | |
622 | ||
623 | @example | |
624 | F list-test | |
625 | (ert-test-failed | |
626 | ((should | |
627 | (equal | |
628 | (list 'a 'b 'c) | |
629 | '(a b d))) | |
630 | :form | |
631 | (equal | |
632 | (a b c) | |
633 | (a b d)) | |
634 | :value nil :explanation | |
635 | (list-elt 2 | |
636 | (different-atoms c d)))) | |
637 | @end example | |
638 | ||
639 | In this case, the function @code{equal} was applied to the arguments | |
640 | @code{(a b c)} and @code{(a b d)}. ERT's explanation shows that | |
641 | the item at index 2 differs between the two lists; in one list, it is | |
642 | the atom c, in the other, it is the atom d. | |
643 | ||
644 | In simple examples like the above, the explanation is unnecessary. | |
645 | But in cases where the difference is not immediately apparent, it can | |
646 | save time: | |
647 | ||
648 | @example | |
649 | F test1 | |
650 | (ert-test-failed | |
651 | ((should | |
652 | (equal x y)) | |
653 | :form | |
654 | (equal a a) | |
655 | :value nil :explanation | |
656 | (different-symbols-with-the-same-name a a))) | |
657 | @end example | |
658 | ||
659 | ERT only provides explanations for predicates that have an explanation | |
660 | function registered. @xref{Defining Explanation Functions}. | |
661 | ||
662 | ||
663 | @node Interactive Debugging, , Understanding Explanations, How to Debug Tests | |
664 | @section Interactive Debugging | |
665 | ||
666 | Debugging failed tests works essentially the same way as debugging any | |
667 | other problems with Lisp code. Here are a few tricks specific to | |
668 | tests: | |
669 | ||
670 | @itemize | |
671 | @item Re-run the failed test a few times to see if it fails in the same way | |
672 | each time. It's good to find out whether the behavior is | |
673 | deterministic before spending any time looking for a cause. In the | |
674 | ERT results buffer, @kbd{r} re-runs the selected test. | |
675 | ||
676 | @item Use @kbd{.} to jump to the source code of the test to find out what | |
677 | exactly it does. Perhaps the test is broken rather than the code | |
678 | under test. | |
679 | ||
680 | @item If the test contains a series of @code{should} forms and you can't | |
681 | tell which one failed, use @kbd{l}, which shows you the list of all | |
682 | @code{should} forms executed during the test before it failed. | |
683 | ||
684 | @item Use @kbd{b} to view the backtrace. You can also use @kbd{d} to re-run | |
685 | the test with debugging enabled, this will enter the debugger and show | |
686 | the backtrace as well; but the top few frames shown there will not be | |
687 | relevant to you since they are ERT's own debugger hook. @kbd{b} | |
688 | strips them out, so it is more convenient. | |
689 | ||
690 | @item If the test or the code under testing prints messages using | |
691 | @code{message}, use @kbd{m} to see what messages it printed before it | |
692 | failed. This can be useful to figure out how far it got. | |
693 | ||
694 | @item You can instrument tests for debugging the same way you instrument | |
695 | @code{defun}s for debugging --- go to the source code of the test and | |
696 | type @kbd{@kbd{C-u} @kbd{C-M-x}}. Then, go back to the ERT buffer and | |
697 | re-run the test with @kbd{r} or @kbd{d}. | |
698 | ||
699 | @item If you have been editing and rearranging tests, it is possible that | |
700 | ERT remembers an old test that you have since renamed or removed --- | |
701 | renamings or removals of definitions in the source code leave around a | |
702 | stray definition under the old name in the running process, this is a | |
703 | common problem in Lisp. In such a situation, hit @kbd{D} to let ERT | |
704 | forget about the obsolete test. | |
705 | @end itemize | |
706 | ||
707 | ||
708 | @node Extending ERT, Other Testing Concepts, How to Debug Tests, Top | |
709 | @chapter Extending ERT | |
710 | ||
711 | There are several ways to add functionality to ERT. | |
712 | ||
713 | @menu | |
714 | * Defining Explanation Functions:: Teach ERT about more predicates. | |
715 | * Low-Level Functions for Working with Tests:: Use ERT's data for your purposes. | |
716 | @end menu | |
717 | ||
718 | ||
719 | @node Defining Explanation Functions, Low-Level Functions for Working with Tests, Extending ERT, Extending ERT | |
720 | @section Defining Explanation Functions | |
721 | ||
722 | The explanation function for a predicate is a function that takes the | |
723 | same arguments as the predicate and returns an @emph{explanation}. | |
724 | The explanation should explain why the predicate, when invoked with | |
725 | the arguments given to the explanation function, returns the value | |
726 | that it returns. The explanation can be any object but should have a | |
727 | comprehensible printed representation. If the return value of the | |
728 | predicate needs no explanation for a given list of arguments, the | |
729 | explanation function should return nil. | |
730 | ||
731 | To associate an explanation function with a predicate, add the | |
732 | property @code{ert-explainer} to the symbol that names the predicate. | |
733 | The value of the property should be the symbol that names the | |
734 | explanation function. | |
735 | ||
736 | ||
737 | @node Low-Level Functions for Working with Tests, , Defining Explanation Functions, Extending ERT | |
738 | @section Low-Level Functions for Working with Tests | |
739 | ||
740 | Both @code{ert-run-tests-interactively} and @code{ert-run-tests-batch} | |
741 | are implemented on top of the lower-level test handling code in the | |
742 | sections named ``Facilities for running a single test'', ``Test | |
743 | selectors'', and ``Facilities for running a whole set of tests''. | |
744 | ||
745 | If you want to write code that works with ERT tests, you should take a | |
746 | look at this lower-level code. Symbols that start with @code{ert--} | |
747 | are internal to ERT, those that start with @code{ert-} but not | |
748 | @code{ert--} are meant to be usable by other code. But there is no | |
749 | mature API yet. | |
750 | ||
751 | Contributions to ERT are welcome. | |
752 | ||
753 | ||
754 | @node Other Testing Concepts, , Extending ERT, Top | |
755 | @chapter Other Testing Concepts | |
756 | ||
757 | For information on mocks, stubs, fixtures, or test suites, see below. | |
758 | ||
759 | ||
760 | @menu | |
761 | * Mocks and Stubs:: Stubbing out code that is irrelevant to the test. | |
762 | * Fixtures and Test Suites:: How ERT differs from tools for other languages. | |
763 | @end menu | |
764 | ||
765 | @node Mocks and Stubs, Fixtures and Test Suites, Other Testing Concepts, Other Testing Concepts | |
766 | @section Other Tools for Emacs Lisp | |
767 | ||
768 | Stubbing out functions or using so-called @emph{mocks} can make it | |
769 | easier to write tests. See | |
770 | @url{http://en.wikipedia.org/wiki/Mock_object} for an explanation of | |
771 | the corresponding concepts in object-oriented languages. | |
772 | ||
773 | ERT does not have built-in support for mocks or stubs. The package | |
774 | @code{el-mock} (see @url{http://www.emacswiki.org/emacs/el-mock.el}) | |
775 | offers mocks for Emacs Lisp and can be used in conjunction with ERT. | |
776 | ||
777 | ||
778 | @node Fixtures and Test Suites, , Mocks and Stubs, Other Testing Concepts | |
779 | @section Fixtures and Test Suites | |
780 | ||
781 | In many ways, ERT is similar to frameworks for other languages like | |
782 | SUnit or JUnit. However, two features commonly found in such | |
783 | frameworks are notably absent from ERT: fixtures and test suites. | |
784 | ||
785 | Fixtures, as used e.g. in SUnit or JUnit, are mainly used to provide | |
786 | an environment for a set of tests, and consist of set-up and tear-down | |
787 | functions. | |
788 | ||
789 | While fixtures are a useful syntactic simplification in other | |
790 | languages, this does not apply to Lisp, where higher-order functions | |
791 | and `unwind-protect' are available. One way to implement and use a | |
792 | fixture in ERT is | |
793 | ||
794 | @lisp | |
795 | (defun my-fixture (body) | |
796 | (unwind-protect | |
797 | (progn [set up] | |
798 | (funcall body)) | |
799 | [tear down])) | |
800 | ||
801 | (ert-deftest my-test () | |
802 | (my-fixture | |
803 | (lambda () | |
804 | [test code]))) | |
805 | @end lisp | |
806 | ||
807 | (Another way would be a @code{with-my-fixture} macro.) This solves | |
808 | the set-up and tear-down part, and additionally allows any test | |
809 | to use any combination of fixtures, so it is more flexible than what | |
810 | other tools typically allow. | |
811 | ||
812 | If the test needs access to the environment the fixture sets up, the | |
813 | fixture can be modified to pass arguments to the body. | |
814 | ||
815 | These are well-known Lisp techniques. Special syntax for them could | |
816 | be added but would provide only a minor simplification. | |
817 | ||
818 | (If you are interested in such syntax, note that splitting set-up and | |
819 | tear-down into separate functions, like *Unit tools usually do, makes | |
820 | it impossible to establish dynamic `let' bindings as part of the | |
821 | fixture. So, blindly imitating the way fixtures are implemented in | |
822 | other languages would be counter-productive in Lisp.) | |
823 | ||
824 | The purpose of test suites is to group related tests together. | |
825 | ||
826 | The most common use of this is to run just the tests for one | |
827 | particular module. Since symbol prefixes are the usual way of | |
828 | separating module namespaces in Emacs Lisp, test selectors already | |
829 | solve this by allowing regexp matching on test names; e.g., the | |
830 | selector "^ert-" selects ERT's self-tests. | |
831 | ||
832 | Other uses include grouping tests by their expected execution time to | |
833 | run quick tests during interactive development and slow tests less | |
834 | frequently. This can be achieved with the @code{:tag} argument to | |
835 | @code{ert-deftest} and @code{tag} test selectors. | |
836 | ||
837 | @bye | |
838 | ||
839 | @c LocalWords: ERT Hagelberg Ohler JUnit namespace docstring ERT's | |
840 | @c LocalWords: backtrace makefiles workflow backtraces API SUnit | |
841 | @c LocalWords: subexpressions |