Commit | Line | Data |
---|---|---|
d221e780 CO |
1 | \input texinfo |
2 | @c %**start of header | |
29993416 | 3 | @setfilename ../../info/ert.info |
d221e780 | 4 | @settitle Emacs Lisp Regression Testing |
c6ab4664 | 5 | @documentencoding UTF-8 |
d221e780 CO |
6 | @c %**end of header |
7 | ||
9e7a4bcf | 8 | @dircategory Emacs misc features |
d221e780 | 9 | @direntry |
f9405d87 | 10 | * ERT: (ert). Emacs Lisp regression testing tool. |
d221e780 CO |
11 | @end direntry |
12 | ||
13 | @copying | |
6bc383b1 | 14 | Copyright @copyright{} 2008, 2010--2014 Free Software Foundation, Inc. |
d221e780 CO |
15 | |
16 | @quotation | |
17 | Permission is granted to copy, distribute and/or modify this document | |
6872a814 | 18 | under the terms of the GNU Free Documentation License, Version 1.3 or |
d221e780 | 19 | any later version published by the Free Software Foundation; with no |
551a89e1 | 20 | Invariant Sections, with the Front-Cover Texts being ``A GNU Manual,'' |
6872a814 | 21 | and with the Back-Cover Texts as in (a) below. A copy of the license |
0b1af106 | 22 | is included in the section entitled ``GNU Free Documentation License''. |
6872a814 GM |
23 | |
24 | (a) The FSF's Back-Cover Text is: ``You have the freedom to copy and | |
6bf430d1 | 25 | modify this GNU manual.'' |
d221e780 CO |
26 | @end quotation |
27 | @end copying | |
28 | ||
33306400 GM |
29 | @titlepage |
30 | @title Emacs Lisp Regression Testing | |
31 | @page | |
32 | @vskip 0pt plus 1filll | |
33 | @insertcopying | |
34 | @end titlepage | |
35 | ||
36 | @contents | |
37 | ||
38 | @ifnottex | |
563a450c | 39 | @node Top |
d221e780 CO |
40 | @top ERT: Emacs Lisp Regression Testing |
41 | ||
33306400 GM |
42 | @insertcopying |
43 | ||
d221e780 CO |
44 | ERT is a tool for automated testing in Emacs Lisp. Its main features |
45 | are facilities for defining tests, running them and reporting the | |
46 | results, and for debugging test failures interactively. | |
47 | ||
48 | ERT is similar to tools for other environments such as JUnit, but has | |
49 | unique features that take advantage of the dynamic and interactive | |
50 | nature of Emacs. Despite its name, it works well both for test-driven | |
51 | development (see | |
52 | @url{http://en.wikipedia.org/wiki/Test-driven_development}) and for | |
53 | traditional software development methods. | |
54 | ||
55 | @menu | |
56 | * Introduction:: A simple example of an ERT test. | |
7359a765 | 57 | * How to Run Tests:: Run tests in Emacs or from the command line. |
d221e780 CO |
58 | * How to Write Tests:: How to add tests to your Emacs Lisp code. |
59 | * How to Debug Tests:: What to do if a test fails. | |
60 | * Extending ERT:: ERT is extensible in several ways. | |
61 | * Other Testing Concepts:: Features not in ERT. | |
0b1af106 | 62 | * GNU Free Documentation License:: The license for this documentation. |
d221e780 CO |
63 | |
64 | @detailmenu | |
65 | --- The Detailed Node Listing --- | |
66 | ||
67 | How to Run Tests | |
68 | ||
69 | * Running Tests Interactively:: Run tests in your current Emacs. | |
70 | * Running Tests in Batch Mode:: Run tests in emacs -Q. | |
7359a765 | 71 | * Test Selectors:: Choose which tests to run. |
d221e780 CO |
72 | |
73 | How to Write Tests | |
74 | ||
7359a765 | 75 | * The @code{should} Macro:: A powerful way to express assertions. |
d221e780 | 76 | * Expected Failures:: Tests for known bugs. |
7359a765 | 77 | * Tests and Their Environment:: Don't depend on customizations; no side effects. |
d221e780 CO |
78 | * Useful Techniques:: Some examples. |
79 | ||
80 | How to Debug Tests | |
81 | ||
82 | * Understanding Explanations:: How ERT gives details on why an assertion failed. | |
83 | * Interactive Debugging:: Tools available in the ERT results buffer. | |
84 | ||
85 | Extending ERT | |
86 | ||
87 | * Defining Explanation Functions:: Teach ERT about more predicates. | |
88 | * Low-Level Functions for Working with Tests:: Use ERT's data for your purposes. | |
89 | ||
90 | Other Testing Concepts | |
91 | ||
466a320e GM |
92 | * Mocks and Stubs:: Stubbing out code that is irrelevant to the test. |
93 | * Fixtures and Test Suites:: How ERT differs from tools for other languages. | |
d221e780 | 94 | |
0b1af106 GM |
95 | Appendix |
96 | ||
97 | * GNU Free Documentation License:: The license for this documentation. | |
98 | ||
d221e780 CO |
99 | @end detailmenu |
100 | @end menu | |
33306400 | 101 | @end ifnottex |
d221e780 | 102 | |
563a450c | 103 | @node Introduction |
d221e780 CO |
104 | @chapter Introduction |
105 | ||
106 | ERT allows you to define @emph{tests} in addition to functions, | |
107 | macros, variables, and the other usual Lisp constructs. Tests are | |
f99f1641 | 108 | simply Lisp code: code that invokes other code and checks whether |
d221e780 CO |
109 | it behaves as expected. |
110 | ||
111 | ERT keeps track of the tests that are defined and provides convenient | |
112 | commands to run them to verify whether the definitions that are | |
113 | currently loaded in Emacs pass the tests. | |
114 | ||
115 | Some Lisp files have comments like the following (adapted from the | |
116 | package @code{pp.el}): | |
117 | ||
118 | @lisp | |
119 | ;; (pp-to-string '(quote quote)) ; expected: "'quote" | |
120 | ;; (pp-to-string '((quote a) (quote b))) ; expected: "('a 'b)\n" | |
121 | ;; (pp-to-string '('a 'b)) ; same as above | |
122 | @end lisp | |
123 | ||
124 | The code contained in these comments can be evaluated from time to | |
125 | time to compare the output with the expected output. ERT formalizes | |
126 | this and introduces a common convention, which simplifies Emacs | |
127 | development, since programmers no longer have to manually find and | |
128 | evaluate such comments. | |
129 | ||
130 | An ERT test definition equivalent to the above comments is this: | |
131 | ||
132 | @lisp | |
133 | (ert-deftest pp-test-quote () | |
134 | "Tests the rendering of `quote' symbols in `pp-to-string'." | |
135 | (should (equal (pp-to-string '(quote quote)) "'quote")) | |
136 | (should (equal (pp-to-string '((quote a) (quote b))) "('a 'b)\n")) | |
137 | (should (equal (pp-to-string '('a 'b)) "('a 'b)\n"))) | |
138 | @end lisp | |
139 | ||
140 | If you know @code{defun}, the syntax of @code{ert-deftest} should look | |
141 | familiar: This example defines a test named @code{pp-test-quote} that | |
4181427f | 142 | will pass if the three calls to @code{equal} all return non-@code{nil}. |
d221e780 | 143 | |
dd90fd1a | 144 | @code{should} is a macro with the same meaning as @code{cl-assert} but |
d221e780 CO |
145 | better error reporting. @xref{The @code{should} Macro}. |
146 | ||
466a320e | 147 | Each test should have a name that describes what functionality it tests. |
f99f1641 PE |
148 | Test names can be chosen arbitrarily---they are in a |
149 | namespace separate from functions and variables---but should follow | |
d221e780 CO |
150 | the usual Emacs Lisp convention of having a prefix that indicates |
151 | which package they belong to. Test names are displayed by ERT when | |
152 | reporting failures and can be used when selecting which tests to run. | |
153 | ||
154 | The empty parentheses @code{()} in the first line don't currently have | |
155 | any meaning and are reserved for future extension. They also make | |
466a320e | 156 | the syntax of @code{ert-deftest} more similar to that of @code{defun}. |
d221e780 CO |
157 | |
158 | The docstring describes what feature this test tests. When running | |
159 | tests interactively, the first line of the docstring is displayed for | |
160 | tests that fail, so it is good if the first line makes sense on its | |
161 | own. | |
162 | ||
163 | The body of a test can be arbitrary Lisp code. It should have as few | |
164 | side effects as possible; each test should be written to clean up | |
165 | after itself, leaving Emacs in the same state as it was before the | |
166 | test. Tests should clean up even if they fail. @xref{Tests and Their | |
167 | Environment}. | |
168 | ||
169 | ||
563a450c | 170 | @node How to Run Tests |
d221e780 CO |
171 | @chapter How to Run Tests |
172 | ||
173 | You can run tests either in the Emacs you are working in, or on the | |
174 | command line in a separate Emacs process in batch mode (i.e., with no | |
175 | user interface). The former mode is convenient during interactive | |
176 | development, the latter is useful to make sure that tests pass | |
466a320e GM |
177 | independently of your customizations; and it allows you to invoke |
178 | tests from makefiles, and to write scripts that run tests in several | |
d221e780 CO |
179 | different Emacs versions. |
180 | ||
181 | @menu | |
182 | * Running Tests Interactively:: Run tests in your current Emacs. | |
183 | * Running Tests in Batch Mode:: Run tests in emacs -Q. | |
466a320e | 184 | * Test Selectors:: Choose which tests to run. |
d221e780 CO |
185 | @end menu |
186 | ||
187 | ||
563a450c | 188 | @node Running Tests Interactively |
d221e780 CO |
189 | @section Running Tests Interactively |
190 | ||
191 | You can run the tests that are currently defined in your Emacs with | |
466a320e GM |
192 | the command @kbd{@kbd{M-x} ert @kbd{RET} t @kbd{RET}}. (For an |
193 | explanation of the @code{t} argument, @pxref{Test Selectors}.) ERT will pop | |
d221e780 CO |
194 | up a new buffer, the ERT results buffer, showing the results of the |
195 | tests run. It looks like this: | |
196 | ||
197 | @example | |
198 | Selector: t | |
955ada11 MA |
199 | Passed: 31 |
200 | Skipped: 0 | |
201 | Failed: 2 (2 unexpected) | |
202 | Total: 33/33 | |
d221e780 CO |
203 | |
204 | Started at: 2008-09-11 08:39:25-0700 | |
205 | Finished. | |
206 | Finished at: 2008-09-11 08:39:27-0700 | |
207 | ||
208 | FF............................... | |
209 | ||
210 | F addition-test | |
211 | (ert-test-failed | |
212 | ((should | |
213 | (= | |
214 | (+ 1 2) | |
215 | 4)) | |
216 | :form | |
217 | (= 3 4) | |
218 | :value nil)) | |
219 | ||
220 | F list-test | |
221 | (ert-test-failed | |
222 | ((should | |
223 | (equal | |
224 | (list 'a 'b 'c) | |
225 | '(a b d))) | |
226 | :form | |
227 | (equal | |
228 | (a b c) | |
229 | (a b d)) | |
230 | :value nil :explanation | |
231 | (list-elt 2 | |
232 | (different-atoms c d)))) | |
233 | @end example | |
234 | ||
466a320e GM |
235 | At the top, there is a summary of the results: we ran all tests defined |
236 | in the current Emacs (@code{Selector: t}), 31 of them passed, and 2 | |
237 | failed unexpectedly. @xref{Expected Failures}, for an explanation of | |
238 | the term @emph{unexpected} in this context. | |
d221e780 CO |
239 | |
240 | The line of dots and @code{F}s is a progress bar where each character | |
241 | represents one test; it fills while the tests are running. A dot | |
242 | means that the test passed, an @code{F} means that it failed. Below | |
243 | the progress bar, ERT shows details about each test that had an | |
244 | unexpected result. In the example above, there are two failures, both | |
245 | due to failed @code{should} forms. @xref{Understanding Explanations}, | |
246 | for more details. | |
247 | ||
248 | In the ERT results buffer, @kbd{TAB} and @kbd{S-TAB} cycle between | |
249 | buttons. Each name of a function or macro in this buffer is a button; | |
250 | moving point to it and typing @kbd{RET} jumps to its definition. | |
251 | ||
252 | Pressing @kbd{r} re-runs the test near point on its own. Pressing | |
253 | @kbd{d} re-runs it with the debugger enabled. @kbd{.} jumps to the | |
254 | definition of the test near point (@kbd{RET} has the same effect if | |
255 | point is on the name of the test). On a failed test, @kbd{b} shows | |
256 | the backtrace of the failure. | |
257 | ||
258 | @kbd{l} shows the list of @code{should} forms executed in the test. | |
259 | If any messages were generated (with the Lisp function @code{message}) | |
260 | in a test or any of the code that it invoked, @kbd{m} will show them. | |
261 | ||
262 | By default, long expressions in the failure details are abbreviated | |
263 | using @code{print-length} and @code{print-level}. Pressing @kbd{L} | |
264 | while point is on a test failure will increase the limits to show more | |
265 | of the expression. | |
266 | ||
267 | ||
563a450c | 268 | @node Running Tests in Batch Mode |
d221e780 CO |
269 | @section Running Tests in Batch Mode |
270 | ||
271 | ERT supports automated invocations from the command line or from | |
272 | scripts or makefiles. There are two functions for this purpose, | |
273 | @code{ert-run-tests-batch} and @code{ert-run-tests-batch-and-exit}. | |
274 | They can be used like this: | |
275 | ||
276 | @example | |
466a320e | 277 | emacs -batch -l ert -l my-tests.el -f ert-run-tests-batch-and-exit |
d221e780 CO |
278 | @end example |
279 | ||
280 | This command will start up Emacs in batch mode, load ERT, load | |
281 | @code{my-tests.el}, and run all tests defined in it. It will exit | |
282 | with a zero exit status if all tests passed, or nonzero if any tests | |
283 | failed or if anything else went wrong. It will also print progress | |
284 | messages and error diagnostics to standard output. | |
285 | ||
466a320e GM |
286 | If ERT is not part of your Emacs distribution, you may need to use |
287 | @code{-L /path/to/ert/} so that Emacs can find it. You may need | |
288 | additional @code{-L} flags to ensure that @code{my-tests.el} and all the | |
289 | files that it requires are on your @code{load-path}. | |
d221e780 CO |
290 | |
291 | ||
563a450c | 292 | @node Test Selectors |
d221e780 CO |
293 | @section Test Selectors |
294 | ||
295 | Functions like @code{ert} accept a @emph{test selector}, a Lisp | |
296 | expression specifying a set of tests. Test selector syntax is similar | |
297 | to Common Lisp's type specifier syntax: | |
298 | ||
299 | @itemize | |
300 | @item @code{nil} selects no tests. | |
301 | @item @code{t} selects all tests. | |
302 | @item @code{:new} selects all tests that have not been run yet. | |
303 | @item @code{:failed} and @code{:passed} select tests according to their most recent result. | |
304 | @item @code{:expected}, @code{:unexpected} select tests according to their most recent result. | |
466a320e GM |
305 | @item A string is a regular expression that selects all tests with matching names. |
306 | @item A test (i.e., an object of @code{ert-test} data type) selects that test. | |
d221e780 | 307 | @item A symbol selects the test that the symbol names. |
466a320e GM |
308 | @item @code{(member TESTS...)} selects the elements of TESTS, a list of |
309 | tests or symbols naming tests. | |
d221e780 CO |
310 | @item @code{(eql TEST)} selects TEST, a test or a symbol naming a test. |
311 | @item @code{(and SELECTORS...)} selects the tests that match all SELECTORS. | |
312 | @item @code{(or SELECTORS...)} selects the tests that match any SELECTOR. | |
313 | @item @code{(not SELECTOR)} selects all tests that do not match SELECTOR. | |
314 | @item @code{(tag TAG)} selects all tests that have TAG on their tags list. | |
466a320e GM |
315 | (Tags are optional labels you can apply to tests when you define them.) |
316 | @item @code{(satisfies PREDICATE)} selects all tests that satisfy PREDICATE, | |
4181427f GM |
317 | a function that takes a test as argument and returns non-@code{nil} if |
318 | it is selected. | |
d221e780 CO |
319 | @end itemize |
320 | ||
321 | Selectors that are frequently useful when selecting tests to run | |
322 | include @code{t} to run all tests that are currently defined in Emacs, | |
466a320e GM |
323 | @code{"^foo-"} to run all tests in package @code{foo} (this assumes |
324 | that package @code{foo} uses the prefix @code{foo-} for its test names), | |
325 | result-based selectors such as @code{(or :new :unexpected)} to | |
d221e780 CO |
326 | run all tests that have either not run yet or that had an unexpected |
327 | result in the last run, and tag-based selectors such as @code{(not | |
328 | (tag :causes-redisplay))} to run all tests that are not tagged | |
329 | @code{:causes-redisplay}. | |
330 | ||
331 | ||
563a450c | 332 | @node How to Write Tests |
d221e780 CO |
333 | @chapter How to Write Tests |
334 | ||
335 | ERT lets you define tests in the same way you define functions. You | |
336 | can type @code{ert-deftest} forms in a buffer and evaluate them there | |
337 | with @code{eval-defun} or @code{compile-defun}, or you can save the | |
338 | file and load it, optionally byte-compiling it first. | |
339 | ||
340 | Just like @code{find-function} is only able to find where a function | |
341 | was defined if the function was loaded from a file, ERT is only able | |
342 | to find where a test was defined if the test was loaded from a file. | |
343 | ||
344 | ||
345 | @menu | |
466a320e | 346 | * The @code{should} Macro:: A powerful way to express assertions. |
d221e780 | 347 | * Expected Failures:: Tests for known bugs. |
466a320e | 348 | * Tests and Their Environment:: Don't depend on customizations; no side effects. |
d221e780 CO |
349 | * Useful Techniques:: Some examples. |
350 | @end menu | |
351 | ||
563a450c | 352 | @node The @code{should} Macro |
d221e780 CO |
353 | @section The @code{should} Macro |
354 | ||
355 | Test bodies can include arbitrary code; but to be useful, they need to | |
466a320e | 356 | check whether the code being tested (or @emph{code under test}) |
d221e780 | 357 | does what it is supposed to do. The macro @code{should} is similar to |
dd90fd1a | 358 | @code{cl-assert} from the cl package |
466a320e GM |
359 | (@pxref{Assertions,,, cl, Common Lisp Extensions}), |
360 | but analyzes its argument form and records information that ERT can | |
361 | display to help debugging. | |
d221e780 CO |
362 | |
363 | This test definition | |
364 | ||
365 | @lisp | |
366 | (ert-deftest addition-test () | |
367 | (should (= (+ 1 2) 4))) | |
368 | @end lisp | |
369 | ||
370 | will produce this output when run via @kbd{M-x ert}: | |
371 | ||
372 | @example | |
373 | F addition-test | |
374 | (ert-test-failed | |
375 | ((should | |
376 | (= | |
377 | (+ 1 2) | |
378 | 4)) | |
379 | :form | |
380 | (= 3 4) | |
381 | :value nil)) | |
382 | @end example | |
383 | ||
384 | In this example, @code{should} recorded the fact that (= (+ 1 2) 4) | |
4181427f | 385 | reduced to (= 3 4) before it reduced to @code{nil}. When debugging why the |
d221e780 CO |
386 | test failed, it helps to know that the function @code{+} returned 3 |
387 | here. ERT records the return value for any predicate called directly | |
388 | within @code{should}. | |
389 | ||
390 | In addition to @code{should}, ERT provides @code{should-not}, which | |
4181427f | 391 | checks that the predicate returns @code{nil}, and @code{should-error}, which |
d221e780 CO |
392 | checks that the form called within it signals an error. An example |
393 | use of @code{should-error}: | |
394 | ||
395 | @lisp | |
396 | (ert-deftest test-divide-by-zero () | |
397 | (should-error (/ 1 0) | |
398 | :type 'arith-error)) | |
399 | @end lisp | |
400 | ||
401 | This checks that dividing one by zero signals an error of type | |
402 | @code{arith-error}. The @code{:type} argument to @code{should-error} | |
403 | is optional; if absent, any type of error is accepted. | |
404 | @code{should-error} returns an error description of the error that was | |
8350f087 | 405 | signaled, to allow additional checks to be made. The error |
d221e780 CO |
406 | description has the format @code{(ERROR-SYMBOL . DATA)}. |
407 | ||
408 | There is no @code{should-not-error} macro since tests that signal an | |
409 | error fail anyway, so @code{should-not-error} is effectively the | |
410 | default. | |
411 | ||
412 | @xref{Understanding Explanations}, for more details on what | |
413 | @code{should} reports. | |
414 | ||
415 | ||
563a450c | 416 | @node Expected Failures |
d221e780 CO |
417 | @section Expected Failures |
418 | ||
466a320e | 419 | Some bugs are complicated to fix, or not very important, and are left as |
d221e780 CO |
420 | @emph{known bugs}. If there is a test case that triggers the bug and |
421 | fails, ERT will alert you of this failure every time you run all | |
422 | tests. For known bugs, this alert is a distraction. The way to | |
423 | suppress it is to add @code{:expected-result :failed} to the test | |
424 | definition: | |
425 | ||
426 | @lisp | |
427 | (ert-deftest future-bug () | |
428 | "Test `time-forward' with negative arguments. | |
466a320e | 429 | Since this functionality isn't implemented, the test is known to fail." |
d221e780 CO |
430 | :expected-result :failed |
431 | (time-forward -1)) | |
432 | @end lisp | |
433 | ||
434 | ERT will still display a small @code{f} in the progress bar as a | |
435 | reminder that there is a known bug, and will count the test as failed, | |
436 | but it will be quiet about it otherwise. | |
437 | ||
438 | An alternative to marking the test as a known failure this way is to | |
439 | delete the test. This is a good idea if there is no intent to fix it, | |
440 | i.e., if the behavior that was formerly considered a bug has become an | |
441 | accepted feature. | |
442 | ||
443 | In general, however, it can be useful to keep tests that are known to | |
444 | fail. If someone wants to fix the bug, they will have a very good | |
445 | starting point: an automated test case that reproduces the bug. This | |
446 | makes it much easier to fix the bug, demonstrate that it is fixed, and | |
447 | prevent future regressions. | |
448 | ||
449 | ERT displays the same kind of alerts for tests that pass unexpectedly | |
466a320e | 450 | as it displays for unexpected failures. This way, if you make code |
d221e780 CO |
451 | changes that happen to fix a bug that you weren't aware of, you will |
452 | know to remove the @code{:expected-result} clause of that test and | |
453 | close the corresponding bug report, if any. | |
454 | ||
455 | Since @code{:expected-result} evaluates its argument when the test is | |
456 | loaded, tests can be marked as known failures only on certain Emacs | |
457 | versions, specific architectures, etc.: | |
458 | ||
459 | @lisp | |
460 | (ert-deftest foo () | |
461 | "A test that is expected to fail on Emacs 23 but succeed elsewhere." | |
462 | :expected-result (if (string-match "GNU Emacs 23[.]" (emacs-version)) | |
463 | :failed | |
464 | :passed) | |
465 | ...) | |
466 | @end lisp | |
467 | ||
468 | ||
563a450c | 469 | @node Tests and Their Environment |
d221e780 CO |
470 | @section Tests and Their Environment |
471 | ||
955ada11 MA |
472 | Sometimes, it doesn't make sense to run a test due to missing |
473 | preconditions. A required Emacs feature might not be compiled in, the | |
474 | function to be tested could call an external binary which might not be | |
475 | available on the test machine, you name it. In this case, the macro | |
476 | @code{skip-unless} could be used to skip the test: | |
477 | ||
478 | @lisp | |
479 | (ert-deftest test-dbus () | |
480 | "A test that checks D-BUS functionality." | |
481 | (skip-unless (featurep 'dbusbind)) | |
482 | ...) | |
483 | @end lisp | |
484 | ||
d221e780 CO |
485 | The outcome of running a test should not depend on the current state |
486 | of the environment, and each test should leave its environment in the | |
487 | same state it found it in. In particular, a test should not depend on | |
488 | any Emacs customization variables or hooks, and if it has to make any | |
466a320e GM |
489 | changes to Emacs's state or state external to Emacs (such as the file |
490 | system), it should undo these changes before it returns, regardless of | |
d221e780 CO |
491 | whether it passed or failed. |
492 | ||
493 | Tests should not depend on the environment because any such | |
494 | dependencies can make the test brittle or lead to failures that occur | |
495 | only under certain circumstances and are hard to reproduce. Of | |
496 | course, the code under test may have settings that affect its | |
497 | behavior. In that case, it is best to make the test @code{let}-bind | |
466a320e | 498 | all such setting variables to set up a specific configuration for the |
d221e780 CO |
499 | duration of the test. The test can also set up a number of different |
500 | configurations and run the code under test with each. | |
501 | ||
502 | Tests that have side effects on their environment should restore it to | |
503 | its original state because any side effects that persist after the | |
504 | test can disrupt the workflow of the programmer running the tests. If | |
466a320e | 505 | the code under test has side effects on Emacs's current state, such as |
d221e780 CO |
506 | on the current buffer or window configuration, the test should create |
507 | a temporary buffer for the code to manipulate (using | |
508 | @code{with-temp-buffer}), or save and restore the window configuration | |
509 | (using @code{save-window-excursion}), respectively. For aspects of | |
510 | the state that can not be preserved with such macros, cleanup should | |
511 | be performed with @code{unwind-protect}, to ensure that the cleanup | |
512 | occurs even if the test fails. | |
513 | ||
514 | An exception to this are messages that the code under test prints with | |
515 | @code{message} and similar logging; tests should not bother restoring | |
d29fbf47 | 516 | the @file{*Message*} buffer to its original state. |
d221e780 CO |
517 | |
518 | The above guidelines imply that tests should avoid calling highly | |
519 | customizable commands such as @code{find-file}, except, of course, if | |
520 | such commands are what they want to test. The exact behavior of | |
521 | @code{find-file} depends on many settings such as | |
522 | @code{find-file-wildcards}, @code{enable-local-variables}, and | |
523 | @code{auto-mode-alist}. It is difficult to write a meaningful test if | |
524 | its behavior can be affected by so many external factors. Also, | |
525 | @code{find-file} has side effects that are hard to predict and thus | |
466a320e | 526 | hard to undo: It may create a new buffer or reuse an existing |
d221e780 CO |
527 | buffer if one is already visiting the requested file; and it runs |
528 | @code{find-file-hook}, which can have arbitrary side effects. | |
529 | ||
530 | Instead, it is better to use lower-level mechanisms with simple and | |
531 | predictable semantics like @code{with-temp-buffer}, @code{insert} or | |
466a320e | 532 | @code{insert-file-contents-literally}, and to activate any desired mode |
f99f1641 | 533 | by calling the corresponding function directly, after binding the |
4181427f | 534 | hook variables to @code{nil}. This avoids the above problems. |
d221e780 CO |
535 | |
536 | ||
563a450c | 537 | @node Useful Techniques |
d221e780 CO |
538 | @section Useful Techniques when Writing Tests |
539 | ||
540 | Testing simple functions that have no side effects and no dependencies | |
541 | on their environment is easy. Such tests often look like this: | |
542 | ||
543 | @lisp | |
544 | (ert-deftest ert-test-mismatch () | |
545 | (should (eql (ert--mismatch "" "") nil)) | |
546 | (should (eql (ert--mismatch "" "a") 0)) | |
547 | (should (eql (ert--mismatch "a" "a") nil)) | |
548 | (should (eql (ert--mismatch "ab" "a") 1)) | |
549 | (should (eql (ert--mismatch "Aa" "aA") 0)) | |
550 | (should (eql (ert--mismatch '(a b c) '(a b d)) 2))) | |
551 | @end lisp | |
552 | ||
553 | This test calls the function @code{ert--mismatch} several times with | |
554 | various combinations of arguments and compares the return value to the | |
555 | expected return value. (Some programmers prefer @code{(should (eql | |
556 | EXPECTED ACTUAL))} over the @code{(should (eql ACTUAL EXPECTED))} | |
557 | shown here. ERT works either way.) | |
558 | ||
559 | Here's a more complicated test: | |
560 | ||
561 | @lisp | |
562 | (ert-deftest ert-test-record-backtrace () | |
563 | (let ((test (make-ert-test :body (lambda () (ert-fail "foo"))))) | |
564 | (let ((result (ert-run-test test))) | |
565 | (should (ert-test-failed-p result)) | |
566 | (with-temp-buffer | |
567 | (ert--print-backtrace (ert-test-failed-backtrace result)) | |
568 | (goto-char (point-min)) | |
569 | (end-of-line) | |
466a320e GM |
570 | (let ((first-line (buffer-substring-no-properties |
571 | (point-min) (point)))) | |
572 | (should (equal first-line | |
573 | " signal(ert-test-failed (\"foo\"))"))))))) | |
d221e780 CO |
574 | @end lisp |
575 | ||
576 | This test creates a test object using @code{make-ert-test} whose body | |
577 | will immediately signal failure. It then runs that test and asserts | |
578 | that it fails. Then, it creates a temporary buffer and invokes | |
579 | @code{ert--print-backtrace} to print the backtrace of the failed test | |
580 | to the current buffer. Finally, it extracts the first line from the | |
581 | buffer and asserts that it matches what we expect. It uses | |
582 | @code{buffer-substring-no-properties} and @code{equal} to ignore text | |
583 | properties; for a test that takes properties into account, | |
584 | @code{buffer-substring} and @code{ert-equal-including-properties} | |
585 | could be used instead. | |
586 | ||
587 | The reason why this test only checks the first line of the backtrace | |
588 | is that the remainder of the backtrace is dependent on ERT's internals | |
589 | as well as whether the code is running interpreted or compiled. By | |
f99f1641 PE |
590 | looking only at the first line, the test checks a useful property---that |
591 | the backtrace correctly captures the call to @code{signal} that | |
592 | results from the call to @code{ert-fail}---without being brittle. | |
d221e780 CO |
593 | |
594 | This example also shows that writing tests is much easier if the code | |
595 | under test was structured with testing in mind. | |
596 | ||
597 | For example, if @code{ert-run-test} accepted only symbols that name | |
598 | tests rather than test objects, the test would need a name for the | |
599 | failing test, which would have to be a temporary symbol generated with | |
466a320e | 600 | @code{make-symbol}, to avoid side effects on Emacs's state. Choosing |
d221e780 CO |
601 | the right interface for @code{ert-run-tests} allows the test to be |
602 | simpler. | |
603 | ||
604 | Similarly, if @code{ert--print-backtrace} printed the backtrace to a | |
605 | buffer with a fixed name rather than the current buffer, it would be | |
606 | much harder for the test to undo the side effect. Of course, some | |
607 | code somewhere needs to pick the buffer name. But that logic is | |
608 | independent of the logic that prints backtraces, and keeping them in | |
609 | separate functions allows us to test them independently. | |
610 | ||
611 | A lot of code that you will encounter in Emacs was not written with | |
612 | testing in mind. Sometimes, the easiest way to write tests for such | |
613 | code is to restructure the code slightly to provide better interfaces | |
614 | for testing. Usually, this makes the interfaces easier to use as | |
615 | well. | |
616 | ||
617 | ||
563a450c | 618 | @node How to Debug Tests |
d221e780 CO |
619 | @chapter How to Debug Tests |
620 | ||
621 | This section describes how to use ERT's features to understand why | |
622 | a test failed. | |
623 | ||
624 | ||
625 | @menu | |
626 | * Understanding Explanations:: How ERT gives details on why an assertion failed. | |
627 | * Interactive Debugging:: Tools available in the ERT results buffer. | |
628 | @end menu | |
629 | ||
630 | ||
563a450c | 631 | @node Understanding Explanations |
d221e780 CO |
632 | @section Understanding Explanations |
633 | ||
634 | Failed @code{should} forms are reported like this: | |
635 | ||
636 | @example | |
637 | F addition-test | |
638 | (ert-test-failed | |
639 | ((should | |
640 | (= | |
641 | (+ 1 2) | |
642 | 4)) | |
643 | :form | |
644 | (= 3 4) | |
645 | :value nil)) | |
646 | @end example | |
647 | ||
648 | ERT shows what the @code{should} expression looked like and what | |
649 | values its subexpressions had: The source code of the assertion was | |
650 | @code{(should (= (+ 1 2) 4))}, which applied the function @code{=} to | |
651 | the arguments @code{3} and @code{4}, resulting in the value | |
652 | @code{nil}. In this case, the test is wrong; it should expect 3 | |
653 | rather than 4. | |
654 | ||
655 | If a predicate like @code{equal} is used with @code{should}, ERT | |
656 | provides a so-called @emph{explanation}: | |
657 | ||
658 | @example | |
659 | F list-test | |
660 | (ert-test-failed | |
661 | ((should | |
662 | (equal | |
663 | (list 'a 'b 'c) | |
664 | '(a b d))) | |
665 | :form | |
666 | (equal | |
667 | (a b c) | |
668 | (a b d)) | |
669 | :value nil :explanation | |
670 | (list-elt 2 | |
671 | (different-atoms c d)))) | |
672 | @end example | |
673 | ||
674 | In this case, the function @code{equal} was applied to the arguments | |
675 | @code{(a b c)} and @code{(a b d)}. ERT's explanation shows that | |
676 | the item at index 2 differs between the two lists; in one list, it is | |
677 | the atom c, in the other, it is the atom d. | |
678 | ||
679 | In simple examples like the above, the explanation is unnecessary. | |
680 | But in cases where the difference is not immediately apparent, it can | |
681 | save time: | |
682 | ||
683 | @example | |
684 | F test1 | |
685 | (ert-test-failed | |
686 | ((should | |
687 | (equal x y)) | |
688 | :form | |
689 | (equal a a) | |
690 | :value nil :explanation | |
691 | (different-symbols-with-the-same-name a a))) | |
692 | @end example | |
693 | ||
694 | ERT only provides explanations for predicates that have an explanation | |
695 | function registered. @xref{Defining Explanation Functions}. | |
696 | ||
697 | ||
563a450c | 698 | @node Interactive Debugging |
d221e780 CO |
699 | @section Interactive Debugging |
700 | ||
466a320e | 701 | Debugging failed tests essentially works the same way as debugging any |
d221e780 CO |
702 | other problems with Lisp code. Here are a few tricks specific to |
703 | tests: | |
704 | ||
705 | @itemize | |
706 | @item Re-run the failed test a few times to see if it fails in the same way | |
707 | each time. It's good to find out whether the behavior is | |
708 | deterministic before spending any time looking for a cause. In the | |
709 | ERT results buffer, @kbd{r} re-runs the selected test. | |
710 | ||
466a320e GM |
711 | @item Use @kbd{.} to jump to the source code of the test to find out exactly |
712 | what it does. Perhaps the test is broken rather than the code | |
d221e780 CO |
713 | under test. |
714 | ||
715 | @item If the test contains a series of @code{should} forms and you can't | |
716 | tell which one failed, use @kbd{l}, which shows you the list of all | |
717 | @code{should} forms executed during the test before it failed. | |
718 | ||
719 | @item Use @kbd{b} to view the backtrace. You can also use @kbd{d} to re-run | |
720 | the test with debugging enabled, this will enter the debugger and show | |
721 | the backtrace as well; but the top few frames shown there will not be | |
722 | relevant to you since they are ERT's own debugger hook. @kbd{b} | |
723 | strips them out, so it is more convenient. | |
724 | ||
725 | @item If the test or the code under testing prints messages using | |
726 | @code{message}, use @kbd{m} to see what messages it printed before it | |
727 | failed. This can be useful to figure out how far it got. | |
728 | ||
729 | @item You can instrument tests for debugging the same way you instrument | |
f99f1641 | 730 | @code{defun}s for debugging: go to the source code of the test and |
d221e780 CO |
731 | type @kbd{@kbd{C-u} @kbd{C-M-x}}. Then, go back to the ERT buffer and |
732 | re-run the test with @kbd{r} or @kbd{d}. | |
733 | ||
734 | @item If you have been editing and rearranging tests, it is possible that | |
f99f1641 | 735 | ERT remembers an old test that you have since renamed or removed: |
d221e780 | 736 | renamings or removals of definitions in the source code leave around a |
466a320e GM |
737 | stray definition under the old name in the running process (this is a |
738 | common problem in Lisp). In such a situation, hit @kbd{D} to let ERT | |
d221e780 CO |
739 | forget about the obsolete test. |
740 | @end itemize | |
741 | ||
742 | ||
563a450c | 743 | @node Extending ERT |
d221e780 CO |
744 | @chapter Extending ERT |
745 | ||
746 | There are several ways to add functionality to ERT. | |
747 | ||
748 | @menu | |
749 | * Defining Explanation Functions:: Teach ERT about more predicates. | |
750 | * Low-Level Functions for Working with Tests:: Use ERT's data for your purposes. | |
751 | @end menu | |
752 | ||
753 | ||
563a450c | 754 | @node Defining Explanation Functions |
d221e780 CO |
755 | @section Defining Explanation Functions |
756 | ||
757 | The explanation function for a predicate is a function that takes the | |
758 | same arguments as the predicate and returns an @emph{explanation}. | |
759 | The explanation should explain why the predicate, when invoked with | |
760 | the arguments given to the explanation function, returns the value | |
761 | that it returns. The explanation can be any object but should have a | |
762 | comprehensible printed representation. If the return value of the | |
763 | predicate needs no explanation for a given list of arguments, the | |
4181427f | 764 | explanation function should return @code{nil}. |
d221e780 CO |
765 | |
766 | To associate an explanation function with a predicate, add the | |
767 | property @code{ert-explainer} to the symbol that names the predicate. | |
768 | The value of the property should be the symbol that names the | |
769 | explanation function. | |
770 | ||
771 | ||
563a450c | 772 | @node Low-Level Functions for Working with Tests |
d221e780 CO |
773 | @section Low-Level Functions for Working with Tests |
774 | ||
775 | Both @code{ert-run-tests-interactively} and @code{ert-run-tests-batch} | |
776 | are implemented on top of the lower-level test handling code in the | |
09e80d9f | 777 | sections of @file{ert.el} labeled ``Facilities for running a single test'', |
466a320e | 778 | ``Test selectors'', and ``Facilities for running a whole set of tests''. |
d221e780 CO |
779 | |
780 | If you want to write code that works with ERT tests, you should take a | |
781 | look at this lower-level code. Symbols that start with @code{ert--} | |
466a320e GM |
782 | are internal to ERT, whereas those that start with @code{ert-} are |
783 | meant to be usable by other code. But there is no mature API yet. | |
d221e780 CO |
784 | |
785 | Contributions to ERT are welcome. | |
786 | ||
787 | ||
563a450c | 788 | @node Other Testing Concepts |
d221e780 CO |
789 | @chapter Other Testing Concepts |
790 | ||
791 | For information on mocks, stubs, fixtures, or test suites, see below. | |
792 | ||
793 | ||
794 | @menu | |
466a320e GM |
795 | * Mocks and Stubs:: Stubbing out code that is irrelevant to the test. |
796 | * Fixtures and Test Suites:: How ERT differs from tools for other languages. | |
d221e780 CO |
797 | @end menu |
798 | ||
563a450c | 799 | @node Mocks and Stubs |
d221e780 CO |
800 | @section Other Tools for Emacs Lisp |
801 | ||
802 | Stubbing out functions or using so-called @emph{mocks} can make it | |
803 | easier to write tests. See | |
804 | @url{http://en.wikipedia.org/wiki/Mock_object} for an explanation of | |
805 | the corresponding concepts in object-oriented languages. | |
806 | ||
807 | ERT does not have built-in support for mocks or stubs. The package | |
808 | @code{el-mock} (see @url{http://www.emacswiki.org/emacs/el-mock.el}) | |
809 | offers mocks for Emacs Lisp and can be used in conjunction with ERT. | |
810 | ||
811 | ||
563a450c | 812 | @node Fixtures and Test Suites |
d221e780 CO |
813 | @section Fixtures and Test Suites |
814 | ||
815 | In many ways, ERT is similar to frameworks for other languages like | |
816 | SUnit or JUnit. However, two features commonly found in such | |
817 | frameworks are notably absent from ERT: fixtures and test suites. | |
818 | ||
466a320e GM |
819 | Fixtures are mainly used (e.g., in SUnit or JUnit) to provide an |
820 | environment for a set of tests, and consist of set-up and tear-down | |
d221e780 CO |
821 | functions. |
822 | ||
823 | While fixtures are a useful syntactic simplification in other | |
824 | languages, this does not apply to Lisp, where higher-order functions | |
825 | and `unwind-protect' are available. One way to implement and use a | |
826 | fixture in ERT is | |
827 | ||
828 | @lisp | |
829 | (defun my-fixture (body) | |
830 | (unwind-protect | |
831 | (progn [set up] | |
832 | (funcall body)) | |
833 | [tear down])) | |
834 | ||
835 | (ert-deftest my-test () | |
836 | (my-fixture | |
837 | (lambda () | |
838 | [test code]))) | |
839 | @end lisp | |
840 | ||
841 | (Another way would be a @code{with-my-fixture} macro.) This solves | |
842 | the set-up and tear-down part, and additionally allows any test | |
843 | to use any combination of fixtures, so it is more flexible than what | |
844 | other tools typically allow. | |
845 | ||
846 | If the test needs access to the environment the fixture sets up, the | |
847 | fixture can be modified to pass arguments to the body. | |
848 | ||
849 | These are well-known Lisp techniques. Special syntax for them could | |
850 | be added but would provide only a minor simplification. | |
851 | ||
852 | (If you are interested in such syntax, note that splitting set-up and | |
853 | tear-down into separate functions, like *Unit tools usually do, makes | |
854 | it impossible to establish dynamic `let' bindings as part of the | |
855 | fixture. So, blindly imitating the way fixtures are implemented in | |
856 | other languages would be counter-productive in Lisp.) | |
857 | ||
858 | The purpose of test suites is to group related tests together. | |
859 | ||
860 | The most common use of this is to run just the tests for one | |
861 | particular module. Since symbol prefixes are the usual way of | |
862 | separating module namespaces in Emacs Lisp, test selectors already | |
863 | solve this by allowing regexp matching on test names; e.g., the | |
864 | selector "^ert-" selects ERT's self-tests. | |
865 | ||
466a320e | 866 | Other uses include grouping tests by their expected execution time, |
1df7defd | 867 | e.g., to run quick tests during interactive development and slow tests less |
466a320e | 868 | often. This can be achieved with the @code{:tag} argument to |
d221e780 CO |
869 | @code{ert-deftest} and @code{tag} test selectors. |
870 | ||
563a450c | 871 | @node GNU Free Documentation License |
0b1af106 GM |
872 | @appendix GNU Free Documentation License |
873 | @include doclicense.texi | |
874 | ||
d221e780 CO |
875 | @bye |
876 | ||
466a320e | 877 | @c LocalWords: ERT JUnit namespace docstring ERT's |
d221e780 CO |
878 | @c LocalWords: backtrace makefiles workflow backtraces API SUnit |
879 | @c LocalWords: subexpressions |