Commit | Line | Data |
---|---|---|
d221e780 CO |
1 | \input texinfo |
2 | @c %**start of header | |
3 | @setfilename ../../info/ert | |
4 | @settitle Emacs Lisp Regression Testing | |
5 | @c %**end of header | |
6 | ||
9e7a4bcf | 7 | @dircategory Emacs misc features |
d221e780 | 8 | @direntry |
9e7a4bcf | 9 | * ERT: (ert). Emacs Lisp regression testing tool. |
d221e780 CO |
10 | @end direntry |
11 | ||
12 | @copying | |
acaf905b | 13 | Copyright @copyright{} 2008, 2010-2012 Free Software Foundation, Inc. |
d221e780 CO |
14 | |
15 | @quotation | |
16 | Permission is granted to copy, distribute and/or modify this document | |
6872a814 | 17 | under the terms of the GNU Free Documentation License, Version 1.3 or |
d221e780 | 18 | any later version published by the Free Software Foundation; with no |
6872a814 GM |
19 | Invariant Sections, with the Front-Cover texts being ``A GNU Manual,'' |
20 | and with the Back-Cover Texts as in (a) below. A copy of the license | |
21 | is included in the section entitled ``GNU Free Documentation License'' | |
22 | in the Emacs manual. | |
23 | ||
24 | (a) The FSF's Back-Cover Text is: ``You have the freedom to copy and | |
25 | modify this GNU manual. Buying copies from the FSF supports it in | |
26 | developing GNU and promoting software freedom.'' | |
27 | ||
28 | This document is part of a collection distributed under the GNU Free | |
29 | Documentation License. If you want to distribute this document | |
30 | separately from the collection, you can do so by adding a copy of the | |
31 | license to the document, as described in section 6 of the license. | |
d221e780 CO |
32 | @end quotation |
33 | @end copying | |
34 | ||
35 | @node Top, Introduction, (dir), (dir) | |
36 | @top ERT: Emacs Lisp Regression Testing | |
37 | ||
38 | ERT is a tool for automated testing in Emacs Lisp. Its main features | |
39 | are facilities for defining tests, running them and reporting the | |
40 | results, and for debugging test failures interactively. | |
41 | ||
42 | ERT is similar to tools for other environments such as JUnit, but has | |
43 | unique features that take advantage of the dynamic and interactive | |
44 | nature of Emacs. Despite its name, it works well both for test-driven | |
45 | development (see | |
46 | @url{http://en.wikipedia.org/wiki/Test-driven_development}) and for | |
47 | traditional software development methods. | |
48 | ||
49 | @menu | |
50 | * Introduction:: A simple example of an ERT test. | |
7359a765 | 51 | * How to Run Tests:: Run tests in Emacs or from the command line. |
d221e780 CO |
52 | * How to Write Tests:: How to add tests to your Emacs Lisp code. |
53 | * How to Debug Tests:: What to do if a test fails. | |
54 | * Extending ERT:: ERT is extensible in several ways. | |
55 | * Other Testing Concepts:: Features not in ERT. | |
56 | ||
57 | @detailmenu | |
58 | --- The Detailed Node Listing --- | |
59 | ||
60 | How to Run Tests | |
61 | ||
62 | * Running Tests Interactively:: Run tests in your current Emacs. | |
63 | * Running Tests in Batch Mode:: Run tests in emacs -Q. | |
7359a765 | 64 | * Test Selectors:: Choose which tests to run. |
d221e780 CO |
65 | |
66 | How to Write Tests | |
67 | ||
7359a765 | 68 | * The @code{should} Macro:: A powerful way to express assertions. |
d221e780 | 69 | * Expected Failures:: Tests for known bugs. |
7359a765 | 70 | * Tests and Their Environment:: Don't depend on customizations; no side effects. |
d221e780 CO |
71 | * Useful Techniques:: Some examples. |
72 | ||
73 | How to Debug Tests | |
74 | ||
75 | * Understanding Explanations:: How ERT gives details on why an assertion failed. | |
76 | * Interactive Debugging:: Tools available in the ERT results buffer. | |
77 | ||
78 | Extending ERT | |
79 | ||
80 | * Defining Explanation Functions:: Teach ERT about more predicates. | |
81 | * Low-Level Functions for Working with Tests:: Use ERT's data for your purposes. | |
82 | ||
83 | Other Testing Concepts | |
84 | ||
466a320e GM |
85 | * Mocks and Stubs:: Stubbing out code that is irrelevant to the test. |
86 | * Fixtures and Test Suites:: How ERT differs from tools for other languages. | |
d221e780 CO |
87 | |
88 | @end detailmenu | |
89 | @end menu | |
90 | ||
91 | @node Introduction, How to Run Tests, Top, Top | |
92 | @chapter Introduction | |
93 | ||
94 | ERT allows you to define @emph{tests} in addition to functions, | |
95 | macros, variables, and the other usual Lisp constructs. Tests are | |
96 | simply Lisp code --- code that invokes other code and checks whether | |
97 | it behaves as expected. | |
98 | ||
99 | ERT keeps track of the tests that are defined and provides convenient | |
100 | commands to run them to verify whether the definitions that are | |
101 | currently loaded in Emacs pass the tests. | |
102 | ||
103 | Some Lisp files have comments like the following (adapted from the | |
104 | package @code{pp.el}): | |
105 | ||
106 | @lisp | |
107 | ;; (pp-to-string '(quote quote)) ; expected: "'quote" | |
108 | ;; (pp-to-string '((quote a) (quote b))) ; expected: "('a 'b)\n" | |
109 | ;; (pp-to-string '('a 'b)) ; same as above | |
110 | @end lisp | |
111 | ||
112 | The code contained in these comments can be evaluated from time to | |
113 | time to compare the output with the expected output. ERT formalizes | |
114 | this and introduces a common convention, which simplifies Emacs | |
115 | development, since programmers no longer have to manually find and | |
116 | evaluate such comments. | |
117 | ||
118 | An ERT test definition equivalent to the above comments is this: | |
119 | ||
120 | @lisp | |
121 | (ert-deftest pp-test-quote () | |
122 | "Tests the rendering of `quote' symbols in `pp-to-string'." | |
123 | (should (equal (pp-to-string '(quote quote)) "'quote")) | |
124 | (should (equal (pp-to-string '((quote a) (quote b))) "('a 'b)\n")) | |
125 | (should (equal (pp-to-string '('a 'b)) "('a 'b)\n"))) | |
126 | @end lisp | |
127 | ||
128 | If you know @code{defun}, the syntax of @code{ert-deftest} should look | |
129 | familiar: This example defines a test named @code{pp-test-quote} that | |
130 | will pass if the three calls to @code{equal} all return true | |
131 | (non-nil). | |
132 | ||
133 | @code{should} is a macro with the same meaning as @code{assert} but | |
134 | better error reporting. @xref{The @code{should} Macro}. | |
135 | ||
466a320e GM |
136 | Each test should have a name that describes what functionality it tests. |
137 | Test names can be chosen arbitrarily --- they are in a | |
d221e780 CO |
138 | namespace separate from functions and variables --- but should follow |
139 | the usual Emacs Lisp convention of having a prefix that indicates | |
140 | which package they belong to. Test names are displayed by ERT when | |
141 | reporting failures and can be used when selecting which tests to run. | |
142 | ||
143 | The empty parentheses @code{()} in the first line don't currently have | |
144 | any meaning and are reserved for future extension. They also make | |
466a320e | 145 | the syntax of @code{ert-deftest} more similar to that of @code{defun}. |
d221e780 CO |
146 | |
147 | The docstring describes what feature this test tests. When running | |
148 | tests interactively, the first line of the docstring is displayed for | |
149 | tests that fail, so it is good if the first line makes sense on its | |
150 | own. | |
151 | ||
152 | The body of a test can be arbitrary Lisp code. It should have as few | |
153 | side effects as possible; each test should be written to clean up | |
154 | after itself, leaving Emacs in the same state as it was before the | |
155 | test. Tests should clean up even if they fail. @xref{Tests and Their | |
156 | Environment}. | |
157 | ||
158 | ||
159 | @node How to Run Tests, How to Write Tests, Introduction, Top | |
160 | @chapter How to Run Tests | |
161 | ||
162 | You can run tests either in the Emacs you are working in, or on the | |
163 | command line in a separate Emacs process in batch mode (i.e., with no | |
164 | user interface). The former mode is convenient during interactive | |
165 | development, the latter is useful to make sure that tests pass | |
466a320e GM |
166 | independently of your customizations; and it allows you to invoke |
167 | tests from makefiles, and to write scripts that run tests in several | |
d221e780 CO |
168 | different Emacs versions. |
169 | ||
170 | @menu | |
171 | * Running Tests Interactively:: Run tests in your current Emacs. | |
172 | * Running Tests in Batch Mode:: Run tests in emacs -Q. | |
466a320e | 173 | * Test Selectors:: Choose which tests to run. |
d221e780 CO |
174 | @end menu |
175 | ||
176 | ||
177 | @node Running Tests Interactively, Running Tests in Batch Mode, How to Run Tests, How to Run Tests | |
178 | @section Running Tests Interactively | |
179 | ||
180 | You can run the tests that are currently defined in your Emacs with | |
466a320e GM |
181 | the command @kbd{@kbd{M-x} ert @kbd{RET} t @kbd{RET}}. (For an |
182 | explanation of the @code{t} argument, @pxref{Test Selectors}.) ERT will pop | |
d221e780 CO |
183 | up a new buffer, the ERT results buffer, showing the results of the |
184 | tests run. It looks like this: | |
185 | ||
186 | @example | |
187 | Selector: t | |
188 | Passed: 31 | |
189 | Failed: 2 (2 unexpected) | |
190 | Total: 33/33 | |
191 | ||
192 | Started at: 2008-09-11 08:39:25-0700 | |
193 | Finished. | |
194 | Finished at: 2008-09-11 08:39:27-0700 | |
195 | ||
196 | FF............................... | |
197 | ||
198 | F addition-test | |
199 | (ert-test-failed | |
200 | ((should | |
201 | (= | |
202 | (+ 1 2) | |
203 | 4)) | |
204 | :form | |
205 | (= 3 4) | |
206 | :value nil)) | |
207 | ||
208 | F list-test | |
209 | (ert-test-failed | |
210 | ((should | |
211 | (equal | |
212 | (list 'a 'b 'c) | |
213 | '(a b d))) | |
214 | :form | |
215 | (equal | |
216 | (a b c) | |
217 | (a b d)) | |
218 | :value nil :explanation | |
219 | (list-elt 2 | |
220 | (different-atoms c d)))) | |
221 | @end example | |
222 | ||
466a320e GM |
223 | At the top, there is a summary of the results: we ran all tests defined |
224 | in the current Emacs (@code{Selector: t}), 31 of them passed, and 2 | |
225 | failed unexpectedly. @xref{Expected Failures}, for an explanation of | |
226 | the term @emph{unexpected} in this context. | |
d221e780 CO |
227 | |
228 | The line of dots and @code{F}s is a progress bar where each character | |
229 | represents one test; it fills while the tests are running. A dot | |
230 | means that the test passed, an @code{F} means that it failed. Below | |
231 | the progress bar, ERT shows details about each test that had an | |
232 | unexpected result. In the example above, there are two failures, both | |
233 | due to failed @code{should} forms. @xref{Understanding Explanations}, | |
234 | for more details. | |
235 | ||
236 | In the ERT results buffer, @kbd{TAB} and @kbd{S-TAB} cycle between | |
237 | buttons. Each name of a function or macro in this buffer is a button; | |
238 | moving point to it and typing @kbd{RET} jumps to its definition. | |
239 | ||
240 | Pressing @kbd{r} re-runs the test near point on its own. Pressing | |
241 | @kbd{d} re-runs it with the debugger enabled. @kbd{.} jumps to the | |
242 | definition of the test near point (@kbd{RET} has the same effect if | |
243 | point is on the name of the test). On a failed test, @kbd{b} shows | |
244 | the backtrace of the failure. | |
245 | ||
246 | @kbd{l} shows the list of @code{should} forms executed in the test. | |
247 | If any messages were generated (with the Lisp function @code{message}) | |
248 | in a test or any of the code that it invoked, @kbd{m} will show them. | |
249 | ||
250 | By default, long expressions in the failure details are abbreviated | |
251 | using @code{print-length} and @code{print-level}. Pressing @kbd{L} | |
252 | while point is on a test failure will increase the limits to show more | |
253 | of the expression. | |
254 | ||
255 | ||
256 | @node Running Tests in Batch Mode, Test Selectors, Running Tests Interactively, How to Run Tests | |
257 | @section Running Tests in Batch Mode | |
258 | ||
259 | ERT supports automated invocations from the command line or from | |
260 | scripts or makefiles. There are two functions for this purpose, | |
261 | @code{ert-run-tests-batch} and @code{ert-run-tests-batch-and-exit}. | |
262 | They can be used like this: | |
263 | ||
264 | @example | |
466a320e | 265 | emacs -batch -l ert -l my-tests.el -f ert-run-tests-batch-and-exit |
d221e780 CO |
266 | @end example |
267 | ||
268 | This command will start up Emacs in batch mode, load ERT, load | |
269 | @code{my-tests.el}, and run all tests defined in it. It will exit | |
270 | with a zero exit status if all tests passed, or nonzero if any tests | |
271 | failed or if anything else went wrong. It will also print progress | |
272 | messages and error diagnostics to standard output. | |
273 | ||
466a320e GM |
274 | If ERT is not part of your Emacs distribution, you may need to use |
275 | @code{-L /path/to/ert/} so that Emacs can find it. You may need | |
276 | additional @code{-L} flags to ensure that @code{my-tests.el} and all the | |
277 | files that it requires are on your @code{load-path}. | |
d221e780 CO |
278 | |
279 | ||
280 | @node Test Selectors, , Running Tests in Batch Mode, How to Run Tests | |
281 | @section Test Selectors | |
282 | ||
283 | Functions like @code{ert} accept a @emph{test selector}, a Lisp | |
284 | expression specifying a set of tests. Test selector syntax is similar | |
285 | to Common Lisp's type specifier syntax: | |
286 | ||
287 | @itemize | |
288 | @item @code{nil} selects no tests. | |
289 | @item @code{t} selects all tests. | |
290 | @item @code{:new} selects all tests that have not been run yet. | |
291 | @item @code{:failed} and @code{:passed} select tests according to their most recent result. | |
292 | @item @code{:expected}, @code{:unexpected} select tests according to their most recent result. | |
466a320e GM |
293 | @item A string is a regular expression that selects all tests with matching names. |
294 | @item A test (i.e., an object of @code{ert-test} data type) selects that test. | |
d221e780 | 295 | @item A symbol selects the test that the symbol names. |
466a320e GM |
296 | @item @code{(member TESTS...)} selects the elements of TESTS, a list of |
297 | tests or symbols naming tests. | |
d221e780 CO |
298 | @item @code{(eql TEST)} selects TEST, a test or a symbol naming a test. |
299 | @item @code{(and SELECTORS...)} selects the tests that match all SELECTORS. | |
300 | @item @code{(or SELECTORS...)} selects the tests that match any SELECTOR. | |
301 | @item @code{(not SELECTOR)} selects all tests that do not match SELECTOR. | |
302 | @item @code{(tag TAG)} selects all tests that have TAG on their tags list. | |
466a320e GM |
303 | (Tags are optional labels you can apply to tests when you define them.) |
304 | @item @code{(satisfies PREDICATE)} selects all tests that satisfy PREDICATE, | |
305 | a function that takes a test as argument and returns non-nil if it is selected. | |
d221e780 CO |
306 | @end itemize |
307 | ||
308 | Selectors that are frequently useful when selecting tests to run | |
309 | include @code{t} to run all tests that are currently defined in Emacs, | |
466a320e GM |
310 | @code{"^foo-"} to run all tests in package @code{foo} (this assumes |
311 | that package @code{foo} uses the prefix @code{foo-} for its test names), | |
312 | result-based selectors such as @code{(or :new :unexpected)} to | |
d221e780 CO |
313 | run all tests that have either not run yet or that had an unexpected |
314 | result in the last run, and tag-based selectors such as @code{(not | |
315 | (tag :causes-redisplay))} to run all tests that are not tagged | |
316 | @code{:causes-redisplay}. | |
317 | ||
318 | ||
319 | @node How to Write Tests, How to Debug Tests, How to Run Tests, Top | |
320 | @chapter How to Write Tests | |
321 | ||
322 | ERT lets you define tests in the same way you define functions. You | |
323 | can type @code{ert-deftest} forms in a buffer and evaluate them there | |
324 | with @code{eval-defun} or @code{compile-defun}, or you can save the | |
325 | file and load it, optionally byte-compiling it first. | |
326 | ||
327 | Just like @code{find-function} is only able to find where a function | |
328 | was defined if the function was loaded from a file, ERT is only able | |
329 | to find where a test was defined if the test was loaded from a file. | |
330 | ||
331 | ||
332 | @menu | |
466a320e | 333 | * The @code{should} Macro:: A powerful way to express assertions. |
d221e780 | 334 | * Expected Failures:: Tests for known bugs. |
466a320e | 335 | * Tests and Their Environment:: Don't depend on customizations; no side effects. |
d221e780 CO |
336 | * Useful Techniques:: Some examples. |
337 | @end menu | |
338 | ||
339 | @node The @code{should} Macro, Expected Failures, How to Write Tests, How to Write Tests | |
340 | @section The @code{should} Macro | |
341 | ||
342 | Test bodies can include arbitrary code; but to be useful, they need to | |
466a320e | 343 | check whether the code being tested (or @emph{code under test}) |
d221e780 | 344 | does what it is supposed to do. The macro @code{should} is similar to |
466a320e GM |
345 | @code{assert} from the cl package |
346 | (@pxref{Assertions,,, cl, Common Lisp Extensions}), | |
347 | but analyzes its argument form and records information that ERT can | |
348 | display to help debugging. | |
d221e780 CO |
349 | |
350 | This test definition | |
351 | ||
352 | @lisp | |
353 | (ert-deftest addition-test () | |
354 | (should (= (+ 1 2) 4))) | |
355 | @end lisp | |
356 | ||
357 | will produce this output when run via @kbd{M-x ert}: | |
358 | ||
359 | @example | |
360 | F addition-test | |
361 | (ert-test-failed | |
362 | ((should | |
363 | (= | |
364 | (+ 1 2) | |
365 | 4)) | |
366 | :form | |
367 | (= 3 4) | |
368 | :value nil)) | |
369 | @end example | |
370 | ||
371 | In this example, @code{should} recorded the fact that (= (+ 1 2) 4) | |
372 | reduced to (= 3 4) before it reduced to nil. When debugging why the | |
373 | test failed, it helps to know that the function @code{+} returned 3 | |
374 | here. ERT records the return value for any predicate called directly | |
375 | within @code{should}. | |
376 | ||
377 | In addition to @code{should}, ERT provides @code{should-not}, which | |
378 | checks that the predicate returns nil, and @code{should-error}, which | |
379 | checks that the form called within it signals an error. An example | |
380 | use of @code{should-error}: | |
381 | ||
382 | @lisp | |
383 | (ert-deftest test-divide-by-zero () | |
384 | (should-error (/ 1 0) | |
385 | :type 'arith-error)) | |
386 | @end lisp | |
387 | ||
388 | This checks that dividing one by zero signals an error of type | |
389 | @code{arith-error}. The @code{:type} argument to @code{should-error} | |
390 | is optional; if absent, any type of error is accepted. | |
391 | @code{should-error} returns an error description of the error that was | |
8350f087 | 392 | signaled, to allow additional checks to be made. The error |
d221e780 CO |
393 | description has the format @code{(ERROR-SYMBOL . DATA)}. |
394 | ||
395 | There is no @code{should-not-error} macro since tests that signal an | |
396 | error fail anyway, so @code{should-not-error} is effectively the | |
397 | default. | |
398 | ||
399 | @xref{Understanding Explanations}, for more details on what | |
400 | @code{should} reports. | |
401 | ||
402 | ||
403 | @node Expected Failures, Tests and Their Environment, The @code{should} Macro, How to Write Tests | |
404 | @section Expected Failures | |
405 | ||
466a320e | 406 | Some bugs are complicated to fix, or not very important, and are left as |
d221e780 CO |
407 | @emph{known bugs}. If there is a test case that triggers the bug and |
408 | fails, ERT will alert you of this failure every time you run all | |
409 | tests. For known bugs, this alert is a distraction. The way to | |
410 | suppress it is to add @code{:expected-result :failed} to the test | |
411 | definition: | |
412 | ||
413 | @lisp | |
414 | (ert-deftest future-bug () | |
415 | "Test `time-forward' with negative arguments. | |
466a320e | 416 | Since this functionality isn't implemented, the test is known to fail." |
d221e780 CO |
417 | :expected-result :failed |
418 | (time-forward -1)) | |
419 | @end lisp | |
420 | ||
421 | ERT will still display a small @code{f} in the progress bar as a | |
422 | reminder that there is a known bug, and will count the test as failed, | |
423 | but it will be quiet about it otherwise. | |
424 | ||
425 | An alternative to marking the test as a known failure this way is to | |
426 | delete the test. This is a good idea if there is no intent to fix it, | |
427 | i.e., if the behavior that was formerly considered a bug has become an | |
428 | accepted feature. | |
429 | ||
430 | In general, however, it can be useful to keep tests that are known to | |
431 | fail. If someone wants to fix the bug, they will have a very good | |
432 | starting point: an automated test case that reproduces the bug. This | |
433 | makes it much easier to fix the bug, demonstrate that it is fixed, and | |
434 | prevent future regressions. | |
435 | ||
436 | ERT displays the same kind of alerts for tests that pass unexpectedly | |
466a320e | 437 | as it displays for unexpected failures. This way, if you make code |
d221e780 CO |
438 | changes that happen to fix a bug that you weren't aware of, you will |
439 | know to remove the @code{:expected-result} clause of that test and | |
440 | close the corresponding bug report, if any. | |
441 | ||
442 | Since @code{:expected-result} evaluates its argument when the test is | |
443 | loaded, tests can be marked as known failures only on certain Emacs | |
444 | versions, specific architectures, etc.: | |
445 | ||
446 | @lisp | |
447 | (ert-deftest foo () | |
448 | "A test that is expected to fail on Emacs 23 but succeed elsewhere." | |
449 | :expected-result (if (string-match "GNU Emacs 23[.]" (emacs-version)) | |
450 | :failed | |
451 | :passed) | |
452 | ...) | |
453 | @end lisp | |
454 | ||
455 | ||
456 | @node Tests and Their Environment, Useful Techniques, Expected Failures, How to Write Tests | |
457 | @section Tests and Their Environment | |
458 | ||
459 | The outcome of running a test should not depend on the current state | |
460 | of the environment, and each test should leave its environment in the | |
461 | same state it found it in. In particular, a test should not depend on | |
462 | any Emacs customization variables or hooks, and if it has to make any | |
466a320e GM |
463 | changes to Emacs's state or state external to Emacs (such as the file |
464 | system), it should undo these changes before it returns, regardless of | |
d221e780 CO |
465 | whether it passed or failed. |
466 | ||
467 | Tests should not depend on the environment because any such | |
468 | dependencies can make the test brittle or lead to failures that occur | |
469 | only under certain circumstances and are hard to reproduce. Of | |
470 | course, the code under test may have settings that affect its | |
471 | behavior. In that case, it is best to make the test @code{let}-bind | |
466a320e | 472 | all such setting variables to set up a specific configuration for the |
d221e780 CO |
473 | duration of the test. The test can also set up a number of different |
474 | configurations and run the code under test with each. | |
475 | ||
476 | Tests that have side effects on their environment should restore it to | |
477 | its original state because any side effects that persist after the | |
478 | test can disrupt the workflow of the programmer running the tests. If | |
466a320e | 479 | the code under test has side effects on Emacs's current state, such as |
d221e780 CO |
480 | on the current buffer or window configuration, the test should create |
481 | a temporary buffer for the code to manipulate (using | |
482 | @code{with-temp-buffer}), or save and restore the window configuration | |
483 | (using @code{save-window-excursion}), respectively. For aspects of | |
484 | the state that can not be preserved with such macros, cleanup should | |
485 | be performed with @code{unwind-protect}, to ensure that the cleanup | |
486 | occurs even if the test fails. | |
487 | ||
488 | An exception to this are messages that the code under test prints with | |
489 | @code{message} and similar logging; tests should not bother restoring | |
490 | the @code{*Message*} buffer to its original state. | |
491 | ||
492 | The above guidelines imply that tests should avoid calling highly | |
493 | customizable commands such as @code{find-file}, except, of course, if | |
494 | such commands are what they want to test. The exact behavior of | |
495 | @code{find-file} depends on many settings such as | |
496 | @code{find-file-wildcards}, @code{enable-local-variables}, and | |
497 | @code{auto-mode-alist}. It is difficult to write a meaningful test if | |
498 | its behavior can be affected by so many external factors. Also, | |
499 | @code{find-file} has side effects that are hard to predict and thus | |
466a320e | 500 | hard to undo: It may create a new buffer or reuse an existing |
d221e780 CO |
501 | buffer if one is already visiting the requested file; and it runs |
502 | @code{find-file-hook}, which can have arbitrary side effects. | |
503 | ||
504 | Instead, it is better to use lower-level mechanisms with simple and | |
505 | predictable semantics like @code{with-temp-buffer}, @code{insert} or | |
466a320e | 506 | @code{insert-file-contents-literally}, and to activate any desired mode |
d221e780 CO |
507 | by calling the corresponding function directly --- after binding the |
508 | hook variables to nil. This avoids the above problems. | |
509 | ||
510 | ||
511 | @node Useful Techniques, , Tests and Their Environment, How to Write Tests | |
512 | @section Useful Techniques when Writing Tests | |
513 | ||
514 | Testing simple functions that have no side effects and no dependencies | |
515 | on their environment is easy. Such tests often look like this: | |
516 | ||
517 | @lisp | |
518 | (ert-deftest ert-test-mismatch () | |
519 | (should (eql (ert--mismatch "" "") nil)) | |
520 | (should (eql (ert--mismatch "" "a") 0)) | |
521 | (should (eql (ert--mismatch "a" "a") nil)) | |
522 | (should (eql (ert--mismatch "ab" "a") 1)) | |
523 | (should (eql (ert--mismatch "Aa" "aA") 0)) | |
524 | (should (eql (ert--mismatch '(a b c) '(a b d)) 2))) | |
525 | @end lisp | |
526 | ||
527 | This test calls the function @code{ert--mismatch} several times with | |
528 | various combinations of arguments and compares the return value to the | |
529 | expected return value. (Some programmers prefer @code{(should (eql | |
530 | EXPECTED ACTUAL))} over the @code{(should (eql ACTUAL EXPECTED))} | |
531 | shown here. ERT works either way.) | |
532 | ||
533 | Here's a more complicated test: | |
534 | ||
535 | @lisp | |
536 | (ert-deftest ert-test-record-backtrace () | |
537 | (let ((test (make-ert-test :body (lambda () (ert-fail "foo"))))) | |
538 | (let ((result (ert-run-test test))) | |
539 | (should (ert-test-failed-p result)) | |
540 | (with-temp-buffer | |
541 | (ert--print-backtrace (ert-test-failed-backtrace result)) | |
542 | (goto-char (point-min)) | |
543 | (end-of-line) | |
466a320e GM |
544 | (let ((first-line (buffer-substring-no-properties |
545 | (point-min) (point)))) | |
546 | (should (equal first-line | |
547 | " signal(ert-test-failed (\"foo\"))"))))))) | |
d221e780 CO |
548 | @end lisp |
549 | ||
550 | This test creates a test object using @code{make-ert-test} whose body | |
551 | will immediately signal failure. It then runs that test and asserts | |
552 | that it fails. Then, it creates a temporary buffer and invokes | |
553 | @code{ert--print-backtrace} to print the backtrace of the failed test | |
554 | to the current buffer. Finally, it extracts the first line from the | |
555 | buffer and asserts that it matches what we expect. It uses | |
556 | @code{buffer-substring-no-properties} and @code{equal} to ignore text | |
557 | properties; for a test that takes properties into account, | |
558 | @code{buffer-substring} and @code{ert-equal-including-properties} | |
559 | could be used instead. | |
560 | ||
561 | The reason why this test only checks the first line of the backtrace | |
562 | is that the remainder of the backtrace is dependent on ERT's internals | |
563 | as well as whether the code is running interpreted or compiled. By | |
564 | looking only at the first line, the test checks a useful property | |
565 | --- that the backtrace correctly captures the call to @code{signal} that | |
566 | results from the call to @code{ert-fail} --- without being brittle. | |
567 | ||
568 | This example also shows that writing tests is much easier if the code | |
569 | under test was structured with testing in mind. | |
570 | ||
571 | For example, if @code{ert-run-test} accepted only symbols that name | |
572 | tests rather than test objects, the test would need a name for the | |
573 | failing test, which would have to be a temporary symbol generated with | |
466a320e | 574 | @code{make-symbol}, to avoid side effects on Emacs's state. Choosing |
d221e780 CO |
575 | the right interface for @code{ert-run-tests} allows the test to be |
576 | simpler. | |
577 | ||
578 | Similarly, if @code{ert--print-backtrace} printed the backtrace to a | |
579 | buffer with a fixed name rather than the current buffer, it would be | |
580 | much harder for the test to undo the side effect. Of course, some | |
581 | code somewhere needs to pick the buffer name. But that logic is | |
582 | independent of the logic that prints backtraces, and keeping them in | |
583 | separate functions allows us to test them independently. | |
584 | ||
585 | A lot of code that you will encounter in Emacs was not written with | |
586 | testing in mind. Sometimes, the easiest way to write tests for such | |
587 | code is to restructure the code slightly to provide better interfaces | |
588 | for testing. Usually, this makes the interfaces easier to use as | |
589 | well. | |
590 | ||
591 | ||
592 | @node How to Debug Tests, Extending ERT, How to Write Tests, Top | |
593 | @chapter How to Debug Tests | |
594 | ||
595 | This section describes how to use ERT's features to understand why | |
596 | a test failed. | |
597 | ||
598 | ||
599 | @menu | |
600 | * Understanding Explanations:: How ERT gives details on why an assertion failed. | |
601 | * Interactive Debugging:: Tools available in the ERT results buffer. | |
602 | @end menu | |
603 | ||
604 | ||
605 | @node Understanding Explanations, Interactive Debugging, How to Debug Tests, How to Debug Tests | |
606 | @section Understanding Explanations | |
607 | ||
608 | Failed @code{should} forms are reported like this: | |
609 | ||
610 | @example | |
611 | F addition-test | |
612 | (ert-test-failed | |
613 | ((should | |
614 | (= | |
615 | (+ 1 2) | |
616 | 4)) | |
617 | :form | |
618 | (= 3 4) | |
619 | :value nil)) | |
620 | @end example | |
621 | ||
622 | ERT shows what the @code{should} expression looked like and what | |
623 | values its subexpressions had: The source code of the assertion was | |
624 | @code{(should (= (+ 1 2) 4))}, which applied the function @code{=} to | |
625 | the arguments @code{3} and @code{4}, resulting in the value | |
626 | @code{nil}. In this case, the test is wrong; it should expect 3 | |
627 | rather than 4. | |
628 | ||
629 | If a predicate like @code{equal} is used with @code{should}, ERT | |
630 | provides a so-called @emph{explanation}: | |
631 | ||
632 | @example | |
633 | F list-test | |
634 | (ert-test-failed | |
635 | ((should | |
636 | (equal | |
637 | (list 'a 'b 'c) | |
638 | '(a b d))) | |
639 | :form | |
640 | (equal | |
641 | (a b c) | |
642 | (a b d)) | |
643 | :value nil :explanation | |
644 | (list-elt 2 | |
645 | (different-atoms c d)))) | |
646 | @end example | |
647 | ||
648 | In this case, the function @code{equal} was applied to the arguments | |
649 | @code{(a b c)} and @code{(a b d)}. ERT's explanation shows that | |
650 | the item at index 2 differs between the two lists; in one list, it is | |
651 | the atom c, in the other, it is the atom d. | |
652 | ||
653 | In simple examples like the above, the explanation is unnecessary. | |
654 | But in cases where the difference is not immediately apparent, it can | |
655 | save time: | |
656 | ||
657 | @example | |
658 | F test1 | |
659 | (ert-test-failed | |
660 | ((should | |
661 | (equal x y)) | |
662 | :form | |
663 | (equal a a) | |
664 | :value nil :explanation | |
665 | (different-symbols-with-the-same-name a a))) | |
666 | @end example | |
667 | ||
668 | ERT only provides explanations for predicates that have an explanation | |
669 | function registered. @xref{Defining Explanation Functions}. | |
670 | ||
671 | ||
672 | @node Interactive Debugging, , Understanding Explanations, How to Debug Tests | |
673 | @section Interactive Debugging | |
674 | ||
466a320e | 675 | Debugging failed tests essentially works the same way as debugging any |
d221e780 CO |
676 | other problems with Lisp code. Here are a few tricks specific to |
677 | tests: | |
678 | ||
679 | @itemize | |
680 | @item Re-run the failed test a few times to see if it fails in the same way | |
681 | each time. It's good to find out whether the behavior is | |
682 | deterministic before spending any time looking for a cause. In the | |
683 | ERT results buffer, @kbd{r} re-runs the selected test. | |
684 | ||
466a320e GM |
685 | @item Use @kbd{.} to jump to the source code of the test to find out exactly |
686 | what it does. Perhaps the test is broken rather than the code | |
d221e780 CO |
687 | under test. |
688 | ||
689 | @item If the test contains a series of @code{should} forms and you can't | |
690 | tell which one failed, use @kbd{l}, which shows you the list of all | |
691 | @code{should} forms executed during the test before it failed. | |
692 | ||
693 | @item Use @kbd{b} to view the backtrace. You can also use @kbd{d} to re-run | |
694 | the test with debugging enabled, this will enter the debugger and show | |
695 | the backtrace as well; but the top few frames shown there will not be | |
696 | relevant to you since they are ERT's own debugger hook. @kbd{b} | |
697 | strips them out, so it is more convenient. | |
698 | ||
699 | @item If the test or the code under testing prints messages using | |
700 | @code{message}, use @kbd{m} to see what messages it printed before it | |
701 | failed. This can be useful to figure out how far it got. | |
702 | ||
703 | @item You can instrument tests for debugging the same way you instrument | |
704 | @code{defun}s for debugging --- go to the source code of the test and | |
705 | type @kbd{@kbd{C-u} @kbd{C-M-x}}. Then, go back to the ERT buffer and | |
706 | re-run the test with @kbd{r} or @kbd{d}. | |
707 | ||
708 | @item If you have been editing and rearranging tests, it is possible that | |
709 | ERT remembers an old test that you have since renamed or removed --- | |
710 | renamings or removals of definitions in the source code leave around a | |
466a320e GM |
711 | stray definition under the old name in the running process (this is a |
712 | common problem in Lisp). In such a situation, hit @kbd{D} to let ERT | |
d221e780 CO |
713 | forget about the obsolete test. |
714 | @end itemize | |
715 | ||
716 | ||
717 | @node Extending ERT, Other Testing Concepts, How to Debug Tests, Top | |
718 | @chapter Extending ERT | |
719 | ||
720 | There are several ways to add functionality to ERT. | |
721 | ||
722 | @menu | |
723 | * Defining Explanation Functions:: Teach ERT about more predicates. | |
724 | * Low-Level Functions for Working with Tests:: Use ERT's data for your purposes. | |
725 | @end menu | |
726 | ||
727 | ||
728 | @node Defining Explanation Functions, Low-Level Functions for Working with Tests, Extending ERT, Extending ERT | |
729 | @section Defining Explanation Functions | |
730 | ||
731 | The explanation function for a predicate is a function that takes the | |
732 | same arguments as the predicate and returns an @emph{explanation}. | |
733 | The explanation should explain why the predicate, when invoked with | |
734 | the arguments given to the explanation function, returns the value | |
735 | that it returns. The explanation can be any object but should have a | |
736 | comprehensible printed representation. If the return value of the | |
737 | predicate needs no explanation for a given list of arguments, the | |
738 | explanation function should return nil. | |
739 | ||
740 | To associate an explanation function with a predicate, add the | |
741 | property @code{ert-explainer} to the symbol that names the predicate. | |
742 | The value of the property should be the symbol that names the | |
743 | explanation function. | |
744 | ||
745 | ||
746 | @node Low-Level Functions for Working with Tests, , Defining Explanation Functions, Extending ERT | |
747 | @section Low-Level Functions for Working with Tests | |
748 | ||
749 | Both @code{ert-run-tests-interactively} and @code{ert-run-tests-batch} | |
750 | are implemented on top of the lower-level test handling code in the | |
09e80d9f | 751 | sections of @file{ert.el} labeled ``Facilities for running a single test'', |
466a320e | 752 | ``Test selectors'', and ``Facilities for running a whole set of tests''. |
d221e780 CO |
753 | |
754 | If you want to write code that works with ERT tests, you should take a | |
755 | look at this lower-level code. Symbols that start with @code{ert--} | |
466a320e GM |
756 | are internal to ERT, whereas those that start with @code{ert-} are |
757 | meant to be usable by other code. But there is no mature API yet. | |
d221e780 CO |
758 | |
759 | Contributions to ERT are welcome. | |
760 | ||
761 | ||
762 | @node Other Testing Concepts, , Extending ERT, Top | |
763 | @chapter Other Testing Concepts | |
764 | ||
765 | For information on mocks, stubs, fixtures, or test suites, see below. | |
766 | ||
767 | ||
768 | @menu | |
466a320e GM |
769 | * Mocks and Stubs:: Stubbing out code that is irrelevant to the test. |
770 | * Fixtures and Test Suites:: How ERT differs from tools for other languages. | |
d221e780 CO |
771 | @end menu |
772 | ||
773 | @node Mocks and Stubs, Fixtures and Test Suites, Other Testing Concepts, Other Testing Concepts | |
774 | @section Other Tools for Emacs Lisp | |
775 | ||
776 | Stubbing out functions or using so-called @emph{mocks} can make it | |
777 | easier to write tests. See | |
778 | @url{http://en.wikipedia.org/wiki/Mock_object} for an explanation of | |
779 | the corresponding concepts in object-oriented languages. | |
780 | ||
781 | ERT does not have built-in support for mocks or stubs. The package | |
782 | @code{el-mock} (see @url{http://www.emacswiki.org/emacs/el-mock.el}) | |
783 | offers mocks for Emacs Lisp and can be used in conjunction with ERT. | |
784 | ||
785 | ||
786 | @node Fixtures and Test Suites, , Mocks and Stubs, Other Testing Concepts | |
787 | @section Fixtures and Test Suites | |
788 | ||
789 | In many ways, ERT is similar to frameworks for other languages like | |
790 | SUnit or JUnit. However, two features commonly found in such | |
791 | frameworks are notably absent from ERT: fixtures and test suites. | |
792 | ||
466a320e GM |
793 | Fixtures are mainly used (e.g., in SUnit or JUnit) to provide an |
794 | environment for a set of tests, and consist of set-up and tear-down | |
d221e780 CO |
795 | functions. |
796 | ||
797 | While fixtures are a useful syntactic simplification in other | |
798 | languages, this does not apply to Lisp, where higher-order functions | |
799 | and `unwind-protect' are available. One way to implement and use a | |
800 | fixture in ERT is | |
801 | ||
802 | @lisp | |
803 | (defun my-fixture (body) | |
804 | (unwind-protect | |
805 | (progn [set up] | |
806 | (funcall body)) | |
807 | [tear down])) | |
808 | ||
809 | (ert-deftest my-test () | |
810 | (my-fixture | |
811 | (lambda () | |
812 | [test code]))) | |
813 | @end lisp | |
814 | ||
815 | (Another way would be a @code{with-my-fixture} macro.) This solves | |
816 | the set-up and tear-down part, and additionally allows any test | |
817 | to use any combination of fixtures, so it is more flexible than what | |
818 | other tools typically allow. | |
819 | ||
820 | If the test needs access to the environment the fixture sets up, the | |
821 | fixture can be modified to pass arguments to the body. | |
822 | ||
823 | These are well-known Lisp techniques. Special syntax for them could | |
824 | be added but would provide only a minor simplification. | |
825 | ||
826 | (If you are interested in such syntax, note that splitting set-up and | |
827 | tear-down into separate functions, like *Unit tools usually do, makes | |
828 | it impossible to establish dynamic `let' bindings as part of the | |
829 | fixture. So, blindly imitating the way fixtures are implemented in | |
830 | other languages would be counter-productive in Lisp.) | |
831 | ||
832 | The purpose of test suites is to group related tests together. | |
833 | ||
834 | The most common use of this is to run just the tests for one | |
835 | particular module. Since symbol prefixes are the usual way of | |
836 | separating module namespaces in Emacs Lisp, test selectors already | |
837 | solve this by allowing regexp matching on test names; e.g., the | |
838 | selector "^ert-" selects ERT's self-tests. | |
839 | ||
466a320e GM |
840 | Other uses include grouping tests by their expected execution time, |
841 | e.g. to run quick tests during interactive development and slow tests less | |
842 | often. This can be achieved with the @code{:tag} argument to | |
d221e780 CO |
843 | @code{ert-deftest} and @code{tag} test selectors. |
844 | ||
845 | @bye | |
846 | ||
466a320e | 847 | @c LocalWords: ERT JUnit namespace docstring ERT's |
d221e780 CO |
848 | @c LocalWords: backtrace makefiles workflow backtraces API SUnit |
849 | @c LocalWords: subexpressions |