Commit | Line | Data |
---|---|---|
d221e780 CO |
1 | \input texinfo |
2 | @c %**start of header | |
3 | @setfilename ../../info/ert | |
4 | @settitle Emacs Lisp Regression Testing | |
5 | @c %**end of header | |
6 | ||
9e7a4bcf | 7 | @dircategory Emacs misc features |
d221e780 | 8 | @direntry |
9e7a4bcf | 9 | * ERT: (ert). Emacs Lisp regression testing tool. |
d221e780 CO |
10 | @end direntry |
11 | ||
12 | @copying | |
acaf905b | 13 | Copyright @copyright{} 2008, 2010-2012 Free Software Foundation, Inc. |
d221e780 CO |
14 | |
15 | @quotation | |
16 | Permission is granted to copy, distribute and/or modify this document | |
6872a814 | 17 | under the terms of the GNU Free Documentation License, Version 1.3 or |
d221e780 | 18 | any later version published by the Free Software Foundation; with no |
6872a814 GM |
19 | Invariant Sections, with the Front-Cover texts being ``A GNU Manual,'' |
20 | and with the Back-Cover Texts as in (a) below. A copy of the license | |
21 | is included in the section entitled ``GNU Free Documentation License'' | |
22 | in the Emacs manual. | |
23 | ||
24 | (a) The FSF's Back-Cover Text is: ``You have the freedom to copy and | |
6bf430d1 | 25 | modify this GNU manual.'' |
6872a814 GM |
26 | |
27 | This document is part of a collection distributed under the GNU Free | |
28 | Documentation License. If you want to distribute this document | |
29 | separately from the collection, you can do so by adding a copy of the | |
30 | license to the document, as described in section 6 of the license. | |
d221e780 CO |
31 | @end quotation |
32 | @end copying | |
33 | ||
34 | @node Top, Introduction, (dir), (dir) | |
35 | @top ERT: Emacs Lisp Regression Testing | |
36 | ||
37 | ERT is a tool for automated testing in Emacs Lisp. Its main features | |
38 | are facilities for defining tests, running them and reporting the | |
39 | results, and for debugging test failures interactively. | |
40 | ||
41 | ERT is similar to tools for other environments such as JUnit, but has | |
42 | unique features that take advantage of the dynamic and interactive | |
43 | nature of Emacs. Despite its name, it works well both for test-driven | |
44 | development (see | |
45 | @url{http://en.wikipedia.org/wiki/Test-driven_development}) and for | |
46 | traditional software development methods. | |
47 | ||
48 | @menu | |
49 | * Introduction:: A simple example of an ERT test. | |
7359a765 | 50 | * How to Run Tests:: Run tests in Emacs or from the command line. |
d221e780 CO |
51 | * How to Write Tests:: How to add tests to your Emacs Lisp code. |
52 | * How to Debug Tests:: What to do if a test fails. | |
53 | * Extending ERT:: ERT is extensible in several ways. | |
54 | * Other Testing Concepts:: Features not in ERT. | |
55 | ||
56 | @detailmenu | |
57 | --- The Detailed Node Listing --- | |
58 | ||
59 | How to Run Tests | |
60 | ||
61 | * Running Tests Interactively:: Run tests in your current Emacs. | |
62 | * Running Tests in Batch Mode:: Run tests in emacs -Q. | |
7359a765 | 63 | * Test Selectors:: Choose which tests to run. |
d221e780 CO |
64 | |
65 | How to Write Tests | |
66 | ||
7359a765 | 67 | * The @code{should} Macro:: A powerful way to express assertions. |
d221e780 | 68 | * Expected Failures:: Tests for known bugs. |
7359a765 | 69 | * Tests and Their Environment:: Don't depend on customizations; no side effects. |
d221e780 CO |
70 | * Useful Techniques:: Some examples. |
71 | ||
72 | How to Debug Tests | |
73 | ||
74 | * Understanding Explanations:: How ERT gives details on why an assertion failed. | |
75 | * Interactive Debugging:: Tools available in the ERT results buffer. | |
76 | ||
77 | Extending ERT | |
78 | ||
79 | * Defining Explanation Functions:: Teach ERT about more predicates. | |
80 | * Low-Level Functions for Working with Tests:: Use ERT's data for your purposes. | |
81 | ||
82 | Other Testing Concepts | |
83 | ||
466a320e GM |
84 | * Mocks and Stubs:: Stubbing out code that is irrelevant to the test. |
85 | * Fixtures and Test Suites:: How ERT differs from tools for other languages. | |
d221e780 CO |
86 | |
87 | @end detailmenu | |
88 | @end menu | |
89 | ||
90 | @node Introduction, How to Run Tests, Top, Top | |
91 | @chapter Introduction | |
92 | ||
93 | ERT allows you to define @emph{tests} in addition to functions, | |
94 | macros, variables, and the other usual Lisp constructs. Tests are | |
95 | simply Lisp code --- code that invokes other code and checks whether | |
96 | it behaves as expected. | |
97 | ||
98 | ERT keeps track of the tests that are defined and provides convenient | |
99 | commands to run them to verify whether the definitions that are | |
100 | currently loaded in Emacs pass the tests. | |
101 | ||
102 | Some Lisp files have comments like the following (adapted from the | |
103 | package @code{pp.el}): | |
104 | ||
105 | @lisp | |
106 | ;; (pp-to-string '(quote quote)) ; expected: "'quote" | |
107 | ;; (pp-to-string '((quote a) (quote b))) ; expected: "('a 'b)\n" | |
108 | ;; (pp-to-string '('a 'b)) ; same as above | |
109 | @end lisp | |
110 | ||
111 | The code contained in these comments can be evaluated from time to | |
112 | time to compare the output with the expected output. ERT formalizes | |
113 | this and introduces a common convention, which simplifies Emacs | |
114 | development, since programmers no longer have to manually find and | |
115 | evaluate such comments. | |
116 | ||
117 | An ERT test definition equivalent to the above comments is this: | |
118 | ||
119 | @lisp | |
120 | (ert-deftest pp-test-quote () | |
121 | "Tests the rendering of `quote' symbols in `pp-to-string'." | |
122 | (should (equal (pp-to-string '(quote quote)) "'quote")) | |
123 | (should (equal (pp-to-string '((quote a) (quote b))) "('a 'b)\n")) | |
124 | (should (equal (pp-to-string '('a 'b)) "('a 'b)\n"))) | |
125 | @end lisp | |
126 | ||
127 | If you know @code{defun}, the syntax of @code{ert-deftest} should look | |
128 | familiar: This example defines a test named @code{pp-test-quote} that | |
129 | will pass if the three calls to @code{equal} all return true | |
130 | (non-nil). | |
131 | ||
dd90fd1a | 132 | @code{should} is a macro with the same meaning as @code{cl-assert} but |
d221e780 CO |
133 | better error reporting. @xref{The @code{should} Macro}. |
134 | ||
466a320e GM |
135 | Each test should have a name that describes what functionality it tests. |
136 | Test names can be chosen arbitrarily --- they are in a | |
d221e780 CO |
137 | namespace separate from functions and variables --- but should follow |
138 | the usual Emacs Lisp convention of having a prefix that indicates | |
139 | which package they belong to. Test names are displayed by ERT when | |
140 | reporting failures and can be used when selecting which tests to run. | |
141 | ||
142 | The empty parentheses @code{()} in the first line don't currently have | |
143 | any meaning and are reserved for future extension. They also make | |
466a320e | 144 | the syntax of @code{ert-deftest} more similar to that of @code{defun}. |
d221e780 CO |
145 | |
146 | The docstring describes what feature this test tests. When running | |
147 | tests interactively, the first line of the docstring is displayed for | |
148 | tests that fail, so it is good if the first line makes sense on its | |
149 | own. | |
150 | ||
151 | The body of a test can be arbitrary Lisp code. It should have as few | |
152 | side effects as possible; each test should be written to clean up | |
153 | after itself, leaving Emacs in the same state as it was before the | |
154 | test. Tests should clean up even if they fail. @xref{Tests and Their | |
155 | Environment}. | |
156 | ||
157 | ||
158 | @node How to Run Tests, How to Write Tests, Introduction, Top | |
159 | @chapter How to Run Tests | |
160 | ||
161 | You can run tests either in the Emacs you are working in, or on the | |
162 | command line in a separate Emacs process in batch mode (i.e., with no | |
163 | user interface). The former mode is convenient during interactive | |
164 | development, the latter is useful to make sure that tests pass | |
466a320e GM |
165 | independently of your customizations; and it allows you to invoke |
166 | tests from makefiles, and to write scripts that run tests in several | |
d221e780 CO |
167 | different Emacs versions. |
168 | ||
169 | @menu | |
170 | * Running Tests Interactively:: Run tests in your current Emacs. | |
171 | * Running Tests in Batch Mode:: Run tests in emacs -Q. | |
466a320e | 172 | * Test Selectors:: Choose which tests to run. |
d221e780 CO |
173 | @end menu |
174 | ||
175 | ||
176 | @node Running Tests Interactively, Running Tests in Batch Mode, How to Run Tests, How to Run Tests | |
177 | @section Running Tests Interactively | |
178 | ||
179 | You can run the tests that are currently defined in your Emacs with | |
466a320e GM |
180 | the command @kbd{@kbd{M-x} ert @kbd{RET} t @kbd{RET}}. (For an |
181 | explanation of the @code{t} argument, @pxref{Test Selectors}.) ERT will pop | |
d221e780 CO |
182 | up a new buffer, the ERT results buffer, showing the results of the |
183 | tests run. It looks like this: | |
184 | ||
185 | @example | |
186 | Selector: t | |
187 | Passed: 31 | |
188 | Failed: 2 (2 unexpected) | |
189 | Total: 33/33 | |
190 | ||
191 | Started at: 2008-09-11 08:39:25-0700 | |
192 | Finished. | |
193 | Finished at: 2008-09-11 08:39:27-0700 | |
194 | ||
195 | FF............................... | |
196 | ||
197 | F addition-test | |
198 | (ert-test-failed | |
199 | ((should | |
200 | (= | |
201 | (+ 1 2) | |
202 | 4)) | |
203 | :form | |
204 | (= 3 4) | |
205 | :value nil)) | |
206 | ||
207 | F list-test | |
208 | (ert-test-failed | |
209 | ((should | |
210 | (equal | |
211 | (list 'a 'b 'c) | |
212 | '(a b d))) | |
213 | :form | |
214 | (equal | |
215 | (a b c) | |
216 | (a b d)) | |
217 | :value nil :explanation | |
218 | (list-elt 2 | |
219 | (different-atoms c d)))) | |
220 | @end example | |
221 | ||
466a320e GM |
222 | At the top, there is a summary of the results: we ran all tests defined |
223 | in the current Emacs (@code{Selector: t}), 31 of them passed, and 2 | |
224 | failed unexpectedly. @xref{Expected Failures}, for an explanation of | |
225 | the term @emph{unexpected} in this context. | |
d221e780 CO |
226 | |
227 | The line of dots and @code{F}s is a progress bar where each character | |
228 | represents one test; it fills while the tests are running. A dot | |
229 | means that the test passed, an @code{F} means that it failed. Below | |
230 | the progress bar, ERT shows details about each test that had an | |
231 | unexpected result. In the example above, there are two failures, both | |
232 | due to failed @code{should} forms. @xref{Understanding Explanations}, | |
233 | for more details. | |
234 | ||
235 | In the ERT results buffer, @kbd{TAB} and @kbd{S-TAB} cycle between | |
236 | buttons. Each name of a function or macro in this buffer is a button; | |
237 | moving point to it and typing @kbd{RET} jumps to its definition. | |
238 | ||
239 | Pressing @kbd{r} re-runs the test near point on its own. Pressing | |
240 | @kbd{d} re-runs it with the debugger enabled. @kbd{.} jumps to the | |
241 | definition of the test near point (@kbd{RET} has the same effect if | |
242 | point is on the name of the test). On a failed test, @kbd{b} shows | |
243 | the backtrace of the failure. | |
244 | ||
245 | @kbd{l} shows the list of @code{should} forms executed in the test. | |
246 | If any messages were generated (with the Lisp function @code{message}) | |
247 | in a test or any of the code that it invoked, @kbd{m} will show them. | |
248 | ||
249 | By default, long expressions in the failure details are abbreviated | |
250 | using @code{print-length} and @code{print-level}. Pressing @kbd{L} | |
251 | while point is on a test failure will increase the limits to show more | |
252 | of the expression. | |
253 | ||
254 | ||
255 | @node Running Tests in Batch Mode, Test Selectors, Running Tests Interactively, How to Run Tests | |
256 | @section Running Tests in Batch Mode | |
257 | ||
258 | ERT supports automated invocations from the command line or from | |
259 | scripts or makefiles. There are two functions for this purpose, | |
260 | @code{ert-run-tests-batch} and @code{ert-run-tests-batch-and-exit}. | |
261 | They can be used like this: | |
262 | ||
263 | @example | |
466a320e | 264 | emacs -batch -l ert -l my-tests.el -f ert-run-tests-batch-and-exit |
d221e780 CO |
265 | @end example |
266 | ||
267 | This command will start up Emacs in batch mode, load ERT, load | |
268 | @code{my-tests.el}, and run all tests defined in it. It will exit | |
269 | with a zero exit status if all tests passed, or nonzero if any tests | |
270 | failed or if anything else went wrong. It will also print progress | |
271 | messages and error diagnostics to standard output. | |
272 | ||
466a320e GM |
273 | If ERT is not part of your Emacs distribution, you may need to use |
274 | @code{-L /path/to/ert/} so that Emacs can find it. You may need | |
275 | additional @code{-L} flags to ensure that @code{my-tests.el} and all the | |
276 | files that it requires are on your @code{load-path}. | |
d221e780 CO |
277 | |
278 | ||
279 | @node Test Selectors, , Running Tests in Batch Mode, How to Run Tests | |
280 | @section Test Selectors | |
281 | ||
282 | Functions like @code{ert} accept a @emph{test selector}, a Lisp | |
283 | expression specifying a set of tests. Test selector syntax is similar | |
284 | to Common Lisp's type specifier syntax: | |
285 | ||
286 | @itemize | |
287 | @item @code{nil} selects no tests. | |
288 | @item @code{t} selects all tests. | |
289 | @item @code{:new} selects all tests that have not been run yet. | |
290 | @item @code{:failed} and @code{:passed} select tests according to their most recent result. | |
291 | @item @code{:expected}, @code{:unexpected} select tests according to their most recent result. | |
466a320e GM |
292 | @item A string is a regular expression that selects all tests with matching names. |
293 | @item A test (i.e., an object of @code{ert-test} data type) selects that test. | |
d221e780 | 294 | @item A symbol selects the test that the symbol names. |
466a320e GM |
295 | @item @code{(member TESTS...)} selects the elements of TESTS, a list of |
296 | tests or symbols naming tests. | |
d221e780 CO |
297 | @item @code{(eql TEST)} selects TEST, a test or a symbol naming a test. |
298 | @item @code{(and SELECTORS...)} selects the tests that match all SELECTORS. | |
299 | @item @code{(or SELECTORS...)} selects the tests that match any SELECTOR. | |
300 | @item @code{(not SELECTOR)} selects all tests that do not match SELECTOR. | |
301 | @item @code{(tag TAG)} selects all tests that have TAG on their tags list. | |
466a320e GM |
302 | (Tags are optional labels you can apply to tests when you define them.) |
303 | @item @code{(satisfies PREDICATE)} selects all tests that satisfy PREDICATE, | |
304 | a function that takes a test as argument and returns non-nil if it is selected. | |
d221e780 CO |
305 | @end itemize |
306 | ||
307 | Selectors that are frequently useful when selecting tests to run | |
308 | include @code{t} to run all tests that are currently defined in Emacs, | |
466a320e GM |
309 | @code{"^foo-"} to run all tests in package @code{foo} (this assumes |
310 | that package @code{foo} uses the prefix @code{foo-} for its test names), | |
311 | result-based selectors such as @code{(or :new :unexpected)} to | |
d221e780 CO |
312 | run all tests that have either not run yet or that had an unexpected |
313 | result in the last run, and tag-based selectors such as @code{(not | |
314 | (tag :causes-redisplay))} to run all tests that are not tagged | |
315 | @code{:causes-redisplay}. | |
316 | ||
317 | ||
318 | @node How to Write Tests, How to Debug Tests, How to Run Tests, Top | |
319 | @chapter How to Write Tests | |
320 | ||
321 | ERT lets you define tests in the same way you define functions. You | |
322 | can type @code{ert-deftest} forms in a buffer and evaluate them there | |
323 | with @code{eval-defun} or @code{compile-defun}, or you can save the | |
324 | file and load it, optionally byte-compiling it first. | |
325 | ||
326 | Just like @code{find-function} is only able to find where a function | |
327 | was defined if the function was loaded from a file, ERT is only able | |
328 | to find where a test was defined if the test was loaded from a file. | |
329 | ||
330 | ||
331 | @menu | |
466a320e | 332 | * The @code{should} Macro:: A powerful way to express assertions. |
d221e780 | 333 | * Expected Failures:: Tests for known bugs. |
466a320e | 334 | * Tests and Their Environment:: Don't depend on customizations; no side effects. |
d221e780 CO |
335 | * Useful Techniques:: Some examples. |
336 | @end menu | |
337 | ||
338 | @node The @code{should} Macro, Expected Failures, How to Write Tests, How to Write Tests | |
339 | @section The @code{should} Macro | |
340 | ||
341 | Test bodies can include arbitrary code; but to be useful, they need to | |
466a320e | 342 | check whether the code being tested (or @emph{code under test}) |
d221e780 | 343 | does what it is supposed to do. The macro @code{should} is similar to |
dd90fd1a | 344 | @code{cl-assert} from the cl package |
466a320e GM |
345 | (@pxref{Assertions,,, cl, Common Lisp Extensions}), |
346 | but analyzes its argument form and records information that ERT can | |
347 | display to help debugging. | |
d221e780 CO |
348 | |
349 | This test definition | |
350 | ||
351 | @lisp | |
352 | (ert-deftest addition-test () | |
353 | (should (= (+ 1 2) 4))) | |
354 | @end lisp | |
355 | ||
356 | will produce this output when run via @kbd{M-x ert}: | |
357 | ||
358 | @example | |
359 | F addition-test | |
360 | (ert-test-failed | |
361 | ((should | |
362 | (= | |
363 | (+ 1 2) | |
364 | 4)) | |
365 | :form | |
366 | (= 3 4) | |
367 | :value nil)) | |
368 | @end example | |
369 | ||
370 | In this example, @code{should} recorded the fact that (= (+ 1 2) 4) | |
371 | reduced to (= 3 4) before it reduced to nil. When debugging why the | |
372 | test failed, it helps to know that the function @code{+} returned 3 | |
373 | here. ERT records the return value for any predicate called directly | |
374 | within @code{should}. | |
375 | ||
376 | In addition to @code{should}, ERT provides @code{should-not}, which | |
377 | checks that the predicate returns nil, and @code{should-error}, which | |
378 | checks that the form called within it signals an error. An example | |
379 | use of @code{should-error}: | |
380 | ||
381 | @lisp | |
382 | (ert-deftest test-divide-by-zero () | |
383 | (should-error (/ 1 0) | |
384 | :type 'arith-error)) | |
385 | @end lisp | |
386 | ||
387 | This checks that dividing one by zero signals an error of type | |
388 | @code{arith-error}. The @code{:type} argument to @code{should-error} | |
389 | is optional; if absent, any type of error is accepted. | |
390 | @code{should-error} returns an error description of the error that was | |
8350f087 | 391 | signaled, to allow additional checks to be made. The error |
d221e780 CO |
392 | description has the format @code{(ERROR-SYMBOL . DATA)}. |
393 | ||
394 | There is no @code{should-not-error} macro since tests that signal an | |
395 | error fail anyway, so @code{should-not-error} is effectively the | |
396 | default. | |
397 | ||
398 | @xref{Understanding Explanations}, for more details on what | |
399 | @code{should} reports. | |
400 | ||
401 | ||
402 | @node Expected Failures, Tests and Their Environment, The @code{should} Macro, How to Write Tests | |
403 | @section Expected Failures | |
404 | ||
466a320e | 405 | Some bugs are complicated to fix, or not very important, and are left as |
d221e780 CO |
406 | @emph{known bugs}. If there is a test case that triggers the bug and |
407 | fails, ERT will alert you of this failure every time you run all | |
408 | tests. For known bugs, this alert is a distraction. The way to | |
409 | suppress it is to add @code{:expected-result :failed} to the test | |
410 | definition: | |
411 | ||
412 | @lisp | |
413 | (ert-deftest future-bug () | |
414 | "Test `time-forward' with negative arguments. | |
466a320e | 415 | Since this functionality isn't implemented, the test is known to fail." |
d221e780 CO |
416 | :expected-result :failed |
417 | (time-forward -1)) | |
418 | @end lisp | |
419 | ||
420 | ERT will still display a small @code{f} in the progress bar as a | |
421 | reminder that there is a known bug, and will count the test as failed, | |
422 | but it will be quiet about it otherwise. | |
423 | ||
424 | An alternative to marking the test as a known failure this way is to | |
425 | delete the test. This is a good idea if there is no intent to fix it, | |
426 | i.e., if the behavior that was formerly considered a bug has become an | |
427 | accepted feature. | |
428 | ||
429 | In general, however, it can be useful to keep tests that are known to | |
430 | fail. If someone wants to fix the bug, they will have a very good | |
431 | starting point: an automated test case that reproduces the bug. This | |
432 | makes it much easier to fix the bug, demonstrate that it is fixed, and | |
433 | prevent future regressions. | |
434 | ||
435 | ERT displays the same kind of alerts for tests that pass unexpectedly | |
466a320e | 436 | as it displays for unexpected failures. This way, if you make code |
d221e780 CO |
437 | changes that happen to fix a bug that you weren't aware of, you will |
438 | know to remove the @code{:expected-result} clause of that test and | |
439 | close the corresponding bug report, if any. | |
440 | ||
441 | Since @code{:expected-result} evaluates its argument when the test is | |
442 | loaded, tests can be marked as known failures only on certain Emacs | |
443 | versions, specific architectures, etc.: | |
444 | ||
445 | @lisp | |
446 | (ert-deftest foo () | |
447 | "A test that is expected to fail on Emacs 23 but succeed elsewhere." | |
448 | :expected-result (if (string-match "GNU Emacs 23[.]" (emacs-version)) | |
449 | :failed | |
450 | :passed) | |
451 | ...) | |
452 | @end lisp | |
453 | ||
454 | ||
455 | @node Tests and Their Environment, Useful Techniques, Expected Failures, How to Write Tests | |
456 | @section Tests and Their Environment | |
457 | ||
458 | The outcome of running a test should not depend on the current state | |
459 | of the environment, and each test should leave its environment in the | |
460 | same state it found it in. In particular, a test should not depend on | |
461 | any Emacs customization variables or hooks, and if it has to make any | |
466a320e GM |
462 | changes to Emacs's state or state external to Emacs (such as the file |
463 | system), it should undo these changes before it returns, regardless of | |
d221e780 CO |
464 | whether it passed or failed. |
465 | ||
466 | Tests should not depend on the environment because any such | |
467 | dependencies can make the test brittle or lead to failures that occur | |
468 | only under certain circumstances and are hard to reproduce. Of | |
469 | course, the code under test may have settings that affect its | |
470 | behavior. In that case, it is best to make the test @code{let}-bind | |
466a320e | 471 | all such setting variables to set up a specific configuration for the |
d221e780 CO |
472 | duration of the test. The test can also set up a number of different |
473 | configurations and run the code under test with each. | |
474 | ||
475 | Tests that have side effects on their environment should restore it to | |
476 | its original state because any side effects that persist after the | |
477 | test can disrupt the workflow of the programmer running the tests. If | |
466a320e | 478 | the code under test has side effects on Emacs's current state, such as |
d221e780 CO |
479 | on the current buffer or window configuration, the test should create |
480 | a temporary buffer for the code to manipulate (using | |
481 | @code{with-temp-buffer}), or save and restore the window configuration | |
482 | (using @code{save-window-excursion}), respectively. For aspects of | |
483 | the state that can not be preserved with such macros, cleanup should | |
484 | be performed with @code{unwind-protect}, to ensure that the cleanup | |
485 | occurs even if the test fails. | |
486 | ||
487 | An exception to this are messages that the code under test prints with | |
488 | @code{message} and similar logging; tests should not bother restoring | |
489 | the @code{*Message*} buffer to its original state. | |
490 | ||
491 | The above guidelines imply that tests should avoid calling highly | |
492 | customizable commands such as @code{find-file}, except, of course, if | |
493 | such commands are what they want to test. The exact behavior of | |
494 | @code{find-file} depends on many settings such as | |
495 | @code{find-file-wildcards}, @code{enable-local-variables}, and | |
496 | @code{auto-mode-alist}. It is difficult to write a meaningful test if | |
497 | its behavior can be affected by so many external factors. Also, | |
498 | @code{find-file} has side effects that are hard to predict and thus | |
466a320e | 499 | hard to undo: It may create a new buffer or reuse an existing |
d221e780 CO |
500 | buffer if one is already visiting the requested file; and it runs |
501 | @code{find-file-hook}, which can have arbitrary side effects. | |
502 | ||
503 | Instead, it is better to use lower-level mechanisms with simple and | |
504 | predictable semantics like @code{with-temp-buffer}, @code{insert} or | |
466a320e | 505 | @code{insert-file-contents-literally}, and to activate any desired mode |
d221e780 CO |
506 | by calling the corresponding function directly --- after binding the |
507 | hook variables to nil. This avoids the above problems. | |
508 | ||
509 | ||
510 | @node Useful Techniques, , Tests and Their Environment, How to Write Tests | |
511 | @section Useful Techniques when Writing Tests | |
512 | ||
513 | Testing simple functions that have no side effects and no dependencies | |
514 | on their environment is easy. Such tests often look like this: | |
515 | ||
516 | @lisp | |
517 | (ert-deftest ert-test-mismatch () | |
518 | (should (eql (ert--mismatch "" "") nil)) | |
519 | (should (eql (ert--mismatch "" "a") 0)) | |
520 | (should (eql (ert--mismatch "a" "a") nil)) | |
521 | (should (eql (ert--mismatch "ab" "a") 1)) | |
522 | (should (eql (ert--mismatch "Aa" "aA") 0)) | |
523 | (should (eql (ert--mismatch '(a b c) '(a b d)) 2))) | |
524 | @end lisp | |
525 | ||
526 | This test calls the function @code{ert--mismatch} several times with | |
527 | various combinations of arguments and compares the return value to the | |
528 | expected return value. (Some programmers prefer @code{(should (eql | |
529 | EXPECTED ACTUAL))} over the @code{(should (eql ACTUAL EXPECTED))} | |
530 | shown here. ERT works either way.) | |
531 | ||
532 | Here's a more complicated test: | |
533 | ||
534 | @lisp | |
535 | (ert-deftest ert-test-record-backtrace () | |
536 | (let ((test (make-ert-test :body (lambda () (ert-fail "foo"))))) | |
537 | (let ((result (ert-run-test test))) | |
538 | (should (ert-test-failed-p result)) | |
539 | (with-temp-buffer | |
540 | (ert--print-backtrace (ert-test-failed-backtrace result)) | |
541 | (goto-char (point-min)) | |
542 | (end-of-line) | |
466a320e GM |
543 | (let ((first-line (buffer-substring-no-properties |
544 | (point-min) (point)))) | |
545 | (should (equal first-line | |
546 | " signal(ert-test-failed (\"foo\"))"))))))) | |
d221e780 CO |
547 | @end lisp |
548 | ||
549 | This test creates a test object using @code{make-ert-test} whose body | |
550 | will immediately signal failure. It then runs that test and asserts | |
551 | that it fails. Then, it creates a temporary buffer and invokes | |
552 | @code{ert--print-backtrace} to print the backtrace of the failed test | |
553 | to the current buffer. Finally, it extracts the first line from the | |
554 | buffer and asserts that it matches what we expect. It uses | |
555 | @code{buffer-substring-no-properties} and @code{equal} to ignore text | |
556 | properties; for a test that takes properties into account, | |
557 | @code{buffer-substring} and @code{ert-equal-including-properties} | |
558 | could be used instead. | |
559 | ||
560 | The reason why this test only checks the first line of the backtrace | |
561 | is that the remainder of the backtrace is dependent on ERT's internals | |
562 | as well as whether the code is running interpreted or compiled. By | |
563 | looking only at the first line, the test checks a useful property | |
564 | --- that the backtrace correctly captures the call to @code{signal} that | |
565 | results from the call to @code{ert-fail} --- without being brittle. | |
566 | ||
567 | This example also shows that writing tests is much easier if the code | |
568 | under test was structured with testing in mind. | |
569 | ||
570 | For example, if @code{ert-run-test} accepted only symbols that name | |
571 | tests rather than test objects, the test would need a name for the | |
572 | failing test, which would have to be a temporary symbol generated with | |
466a320e | 573 | @code{make-symbol}, to avoid side effects on Emacs's state. Choosing |
d221e780 CO |
574 | the right interface for @code{ert-run-tests} allows the test to be |
575 | simpler. | |
576 | ||
577 | Similarly, if @code{ert--print-backtrace} printed the backtrace to a | |
578 | buffer with a fixed name rather than the current buffer, it would be | |
579 | much harder for the test to undo the side effect. Of course, some | |
580 | code somewhere needs to pick the buffer name. But that logic is | |
581 | independent of the logic that prints backtraces, and keeping them in | |
582 | separate functions allows us to test them independently. | |
583 | ||
584 | A lot of code that you will encounter in Emacs was not written with | |
585 | testing in mind. Sometimes, the easiest way to write tests for such | |
586 | code is to restructure the code slightly to provide better interfaces | |
587 | for testing. Usually, this makes the interfaces easier to use as | |
588 | well. | |
589 | ||
590 | ||
591 | @node How to Debug Tests, Extending ERT, How to Write Tests, Top | |
592 | @chapter How to Debug Tests | |
593 | ||
594 | This section describes how to use ERT's features to understand why | |
595 | a test failed. | |
596 | ||
597 | ||
598 | @menu | |
599 | * Understanding Explanations:: How ERT gives details on why an assertion failed. | |
600 | * Interactive Debugging:: Tools available in the ERT results buffer. | |
601 | @end menu | |
602 | ||
603 | ||
604 | @node Understanding Explanations, Interactive Debugging, How to Debug Tests, How to Debug Tests | |
605 | @section Understanding Explanations | |
606 | ||
607 | Failed @code{should} forms are reported like this: | |
608 | ||
609 | @example | |
610 | F addition-test | |
611 | (ert-test-failed | |
612 | ((should | |
613 | (= | |
614 | (+ 1 2) | |
615 | 4)) | |
616 | :form | |
617 | (= 3 4) | |
618 | :value nil)) | |
619 | @end example | |
620 | ||
621 | ERT shows what the @code{should} expression looked like and what | |
622 | values its subexpressions had: The source code of the assertion was | |
623 | @code{(should (= (+ 1 2) 4))}, which applied the function @code{=} to | |
624 | the arguments @code{3} and @code{4}, resulting in the value | |
625 | @code{nil}. In this case, the test is wrong; it should expect 3 | |
626 | rather than 4. | |
627 | ||
628 | If a predicate like @code{equal} is used with @code{should}, ERT | |
629 | provides a so-called @emph{explanation}: | |
630 | ||
631 | @example | |
632 | F list-test | |
633 | (ert-test-failed | |
634 | ((should | |
635 | (equal | |
636 | (list 'a 'b 'c) | |
637 | '(a b d))) | |
638 | :form | |
639 | (equal | |
640 | (a b c) | |
641 | (a b d)) | |
642 | :value nil :explanation | |
643 | (list-elt 2 | |
644 | (different-atoms c d)))) | |
645 | @end example | |
646 | ||
647 | In this case, the function @code{equal} was applied to the arguments | |
648 | @code{(a b c)} and @code{(a b d)}. ERT's explanation shows that | |
649 | the item at index 2 differs between the two lists; in one list, it is | |
650 | the atom c, in the other, it is the atom d. | |
651 | ||
652 | In simple examples like the above, the explanation is unnecessary. | |
653 | But in cases where the difference is not immediately apparent, it can | |
654 | save time: | |
655 | ||
656 | @example | |
657 | F test1 | |
658 | (ert-test-failed | |
659 | ((should | |
660 | (equal x y)) | |
661 | :form | |
662 | (equal a a) | |
663 | :value nil :explanation | |
664 | (different-symbols-with-the-same-name a a))) | |
665 | @end example | |
666 | ||
667 | ERT only provides explanations for predicates that have an explanation | |
668 | function registered. @xref{Defining Explanation Functions}. | |
669 | ||
670 | ||
671 | @node Interactive Debugging, , Understanding Explanations, How to Debug Tests | |
672 | @section Interactive Debugging | |
673 | ||
466a320e | 674 | Debugging failed tests essentially works the same way as debugging any |
d221e780 CO |
675 | other problems with Lisp code. Here are a few tricks specific to |
676 | tests: | |
677 | ||
678 | @itemize | |
679 | @item Re-run the failed test a few times to see if it fails in the same way | |
680 | each time. It's good to find out whether the behavior is | |
681 | deterministic before spending any time looking for a cause. In the | |
682 | ERT results buffer, @kbd{r} re-runs the selected test. | |
683 | ||
466a320e GM |
684 | @item Use @kbd{.} to jump to the source code of the test to find out exactly |
685 | what it does. Perhaps the test is broken rather than the code | |
d221e780 CO |
686 | under test. |
687 | ||
688 | @item If the test contains a series of @code{should} forms and you can't | |
689 | tell which one failed, use @kbd{l}, which shows you the list of all | |
690 | @code{should} forms executed during the test before it failed. | |
691 | ||
692 | @item Use @kbd{b} to view the backtrace. You can also use @kbd{d} to re-run | |
693 | the test with debugging enabled, this will enter the debugger and show | |
694 | the backtrace as well; but the top few frames shown there will not be | |
695 | relevant to you since they are ERT's own debugger hook. @kbd{b} | |
696 | strips them out, so it is more convenient. | |
697 | ||
698 | @item If the test or the code under testing prints messages using | |
699 | @code{message}, use @kbd{m} to see what messages it printed before it | |
700 | failed. This can be useful to figure out how far it got. | |
701 | ||
702 | @item You can instrument tests for debugging the same way you instrument | |
703 | @code{defun}s for debugging --- go to the source code of the test and | |
704 | type @kbd{@kbd{C-u} @kbd{C-M-x}}. Then, go back to the ERT buffer and | |
705 | re-run the test with @kbd{r} or @kbd{d}. | |
706 | ||
707 | @item If you have been editing and rearranging tests, it is possible that | |
708 | ERT remembers an old test that you have since renamed or removed --- | |
709 | renamings or removals of definitions in the source code leave around a | |
466a320e GM |
710 | stray definition under the old name in the running process (this is a |
711 | common problem in Lisp). In such a situation, hit @kbd{D} to let ERT | |
d221e780 CO |
712 | forget about the obsolete test. |
713 | @end itemize | |
714 | ||
715 | ||
716 | @node Extending ERT, Other Testing Concepts, How to Debug Tests, Top | |
717 | @chapter Extending ERT | |
718 | ||
719 | There are several ways to add functionality to ERT. | |
720 | ||
721 | @menu | |
722 | * Defining Explanation Functions:: Teach ERT about more predicates. | |
723 | * Low-Level Functions for Working with Tests:: Use ERT's data for your purposes. | |
724 | @end menu | |
725 | ||
726 | ||
727 | @node Defining Explanation Functions, Low-Level Functions for Working with Tests, Extending ERT, Extending ERT | |
728 | @section Defining Explanation Functions | |
729 | ||
730 | The explanation function for a predicate is a function that takes the | |
731 | same arguments as the predicate and returns an @emph{explanation}. | |
732 | The explanation should explain why the predicate, when invoked with | |
733 | the arguments given to the explanation function, returns the value | |
734 | that it returns. The explanation can be any object but should have a | |
735 | comprehensible printed representation. If the return value of the | |
736 | predicate needs no explanation for a given list of arguments, the | |
737 | explanation function should return nil. | |
738 | ||
739 | To associate an explanation function with a predicate, add the | |
740 | property @code{ert-explainer} to the symbol that names the predicate. | |
741 | The value of the property should be the symbol that names the | |
742 | explanation function. | |
743 | ||
744 | ||
745 | @node Low-Level Functions for Working with Tests, , Defining Explanation Functions, Extending ERT | |
746 | @section Low-Level Functions for Working with Tests | |
747 | ||
748 | Both @code{ert-run-tests-interactively} and @code{ert-run-tests-batch} | |
749 | are implemented on top of the lower-level test handling code in the | |
09e80d9f | 750 | sections of @file{ert.el} labeled ``Facilities for running a single test'', |
466a320e | 751 | ``Test selectors'', and ``Facilities for running a whole set of tests''. |
d221e780 CO |
752 | |
753 | If you want to write code that works with ERT tests, you should take a | |
754 | look at this lower-level code. Symbols that start with @code{ert--} | |
466a320e GM |
755 | are internal to ERT, whereas those that start with @code{ert-} are |
756 | meant to be usable by other code. But there is no mature API yet. | |
d221e780 CO |
757 | |
758 | Contributions to ERT are welcome. | |
759 | ||
760 | ||
761 | @node Other Testing Concepts, , Extending ERT, Top | |
762 | @chapter Other Testing Concepts | |
763 | ||
764 | For information on mocks, stubs, fixtures, or test suites, see below. | |
765 | ||
766 | ||
767 | @menu | |
466a320e GM |
768 | * Mocks and Stubs:: Stubbing out code that is irrelevant to the test. |
769 | * Fixtures and Test Suites:: How ERT differs from tools for other languages. | |
d221e780 CO |
770 | @end menu |
771 | ||
772 | @node Mocks and Stubs, Fixtures and Test Suites, Other Testing Concepts, Other Testing Concepts | |
773 | @section Other Tools for Emacs Lisp | |
774 | ||
775 | Stubbing out functions or using so-called @emph{mocks} can make it | |
776 | easier to write tests. See | |
777 | @url{http://en.wikipedia.org/wiki/Mock_object} for an explanation of | |
778 | the corresponding concepts in object-oriented languages. | |
779 | ||
780 | ERT does not have built-in support for mocks or stubs. The package | |
781 | @code{el-mock} (see @url{http://www.emacswiki.org/emacs/el-mock.el}) | |
782 | offers mocks for Emacs Lisp and can be used in conjunction with ERT. | |
783 | ||
784 | ||
785 | @node Fixtures and Test Suites, , Mocks and Stubs, Other Testing Concepts | |
786 | @section Fixtures and Test Suites | |
787 | ||
788 | In many ways, ERT is similar to frameworks for other languages like | |
789 | SUnit or JUnit. However, two features commonly found in such | |
790 | frameworks are notably absent from ERT: fixtures and test suites. | |
791 | ||
466a320e GM |
792 | Fixtures are mainly used (e.g., in SUnit or JUnit) to provide an |
793 | environment for a set of tests, and consist of set-up and tear-down | |
d221e780 CO |
794 | functions. |
795 | ||
796 | While fixtures are a useful syntactic simplification in other | |
797 | languages, this does not apply to Lisp, where higher-order functions | |
798 | and `unwind-protect' are available. One way to implement and use a | |
799 | fixture in ERT is | |
800 | ||
801 | @lisp | |
802 | (defun my-fixture (body) | |
803 | (unwind-protect | |
804 | (progn [set up] | |
805 | (funcall body)) | |
806 | [tear down])) | |
807 | ||
808 | (ert-deftest my-test () | |
809 | (my-fixture | |
810 | (lambda () | |
811 | [test code]))) | |
812 | @end lisp | |
813 | ||
814 | (Another way would be a @code{with-my-fixture} macro.) This solves | |
815 | the set-up and tear-down part, and additionally allows any test | |
816 | to use any combination of fixtures, so it is more flexible than what | |
817 | other tools typically allow. | |
818 | ||
819 | If the test needs access to the environment the fixture sets up, the | |
820 | fixture can be modified to pass arguments to the body. | |
821 | ||
822 | These are well-known Lisp techniques. Special syntax for them could | |
823 | be added but would provide only a minor simplification. | |
824 | ||
825 | (If you are interested in such syntax, note that splitting set-up and | |
826 | tear-down into separate functions, like *Unit tools usually do, makes | |
827 | it impossible to establish dynamic `let' bindings as part of the | |
828 | fixture. So, blindly imitating the way fixtures are implemented in | |
829 | other languages would be counter-productive in Lisp.) | |
830 | ||
831 | The purpose of test suites is to group related tests together. | |
832 | ||
833 | The most common use of this is to run just the tests for one | |
834 | particular module. Since symbol prefixes are the usual way of | |
835 | separating module namespaces in Emacs Lisp, test selectors already | |
836 | solve this by allowing regexp matching on test names; e.g., the | |
837 | selector "^ert-" selects ERT's self-tests. | |
838 | ||
466a320e | 839 | Other uses include grouping tests by their expected execution time, |
1df7defd | 840 | e.g., to run quick tests during interactive development and slow tests less |
466a320e | 841 | often. This can be achieved with the @code{:tag} argument to |
d221e780 CO |
842 | @code{ert-deftest} and @code{tag} test selectors. |
843 | ||
844 | @bye | |
845 | ||
466a320e | 846 | @c LocalWords: ERT JUnit namespace docstring ERT's |
d221e780 CO |
847 | @c LocalWords: backtrace makefiles workflow backtraces API SUnit |
848 | @c LocalWords: subexpressions |