(Fcall_process): Use system_eol_type for encoding
[bpt/emacs.git] / lispref / nonascii.texi
CommitLineData
cc6d0d2c
RS
1@c -*-texinfo-*-
2@c This is part of the GNU Emacs Lisp Reference Manual.
651f374c 3@c Copyright (C) 1998, 1999, 2002, 2003, 2004,
ceb4c4d3 4@c 2005, 2006 Free Software Foundation, Inc.
cc6d0d2c
RS
5@c See the file elisp.texi for copying conditions.
6@setfilename ../info/characters
7@node Non-ASCII Characters, Searching and Matching, Text, Top
ad800164 8@chapter Non-@acronym{ASCII} Characters
cc6d0d2c 9@cindex multibyte characters
ad800164 10@cindex non-@acronym{ASCII} characters
cc6d0d2c 11
ad800164 12 This chapter covers the special issues relating to non-@acronym{ASCII}
cc6d0d2c
RS
13characters and how they are stored in strings and buffers.
14
15@menu
5557b83b
RS
16* Text Representations:: Unibyte and multibyte representations
17* Converting Representations:: Converting unibyte to multibyte and vice versa.
18* Selecting a Representation:: Treating a byte sequence as unibyte or multi.
19* Character Codes:: How unibyte and multibyte relate to
20 codes of individual characters.
a3d3f60d 21* Character Sets:: The space of possible character codes
5557b83b
RS
22 is divided into various character sets.
23* Chars and Bytes:: More information about multibyte encodings.
24* Splitting Characters:: Converting a character to its byte sequence.
25* Scanning Charsets:: Which character sets are used in a buffer?
26* Translation of Characters:: Translation tables are used for conversion.
27* Coding Systems:: Coding systems are conversions for saving files.
28* Input Methods:: Input methods allow users to enter various
8a9e355c 29 non-ASCII characters without special keyboards.
5557b83b 30* Locales:: Interacting with the POSIX locale.
cc6d0d2c
RS
31@end menu
32
33@node Text Representations
34@section Text Representations
35@cindex text representations
36
37 Emacs has two @dfn{text representations}---two ways to represent text
38in a string or buffer. These are called @dfn{unibyte} and
39@dfn{multibyte}. Each string, and each buffer, uses one of these two
40representations. For most purposes, you can ignore the issue of
41representations, because Emacs converts text between them as
42appropriate. Occasionally in Lisp programming you will need to pay
43attention to the difference.
44
45@cindex unibyte text
46 In unibyte representation, each character occupies one byte and
47therefore the possible character codes range from 0 to 255. Codes 0
ad800164
EZ
48through 127 are @acronym{ASCII} characters; the codes from 128 through 255
49are used for one non-@acronym{ASCII} character set (you can choose which
969fe9b5 50character set by setting the variable @code{nonascii-insert-offset}).
cc6d0d2c
RS
51
52@cindex leading code
53@cindex multibyte text
1911e6e5 54@cindex trailing codes
cc6d0d2c
RS
55 In multibyte representation, a character may occupy more than one
56byte, and as a result, the full range of Emacs character codes can be
57stored. The first byte of a multibyte character is always in the range
58128 through 159 (octal 0200 through 0237). These values are called
a9f0a989
RS
59@dfn{leading codes}. The second and subsequent bytes of a multibyte
60character are always in the range 160 through 255 (octal 0240 through
1911e6e5 610377); these values are @dfn{trailing codes}.
cc6d0d2c 62
0ace421a 63 Some sequences of bytes are not valid in multibyte text: for example,
1e4d32f8
GM
64a single isolated byte in the range 128 through 159 is not allowed. But
65character codes 128 through 159 can appear in multibyte text,
66represented as two-byte sequences. All the character codes 128 through
67255 are possible (though slightly abnormal) in multibyte text; they
0ace421a
GM
68appear in multibyte buffers and strings when you do explicit encoding
69and decoding (@pxref{Explicit Encoding}).
b6954afd 70
cc6d0d2c
RS
71 In a buffer, the buffer-local value of the variable
72@code{enable-multibyte-characters} specifies the representation used.
08f0f5e9
KH
73The representation for a string is determined and recorded in the string
74when the string is constructed.
cc6d0d2c 75
cc6d0d2c
RS
76@defvar enable-multibyte-characters
77This variable specifies the current buffer's text representation.
78If it is non-@code{nil}, the buffer contains multibyte text; otherwise,
79it contains unibyte text.
80
969fe9b5
RS
81You cannot set this variable directly; instead, use the function
82@code{set-buffer-multibyte} to change a buffer's representation.
cc6d0d2c
RS
83@end defvar
84
cc6d0d2c 85@defvar default-enable-multibyte-characters
a9f0a989 86This variable's value is entirely equivalent to @code{(default-value
cc6d0d2c 87'enable-multibyte-characters)}, and setting this variable changes that
a9f0a989
RS
88default value. Setting the local binding of
89@code{enable-multibyte-characters} in a specific buffer is not allowed,
90but changing the default value is supported, and it is a reasonable
91thing to do, because it has no effect on existing buffers.
cc6d0d2c
RS
92
93The @samp{--unibyte} command line option does its job by setting the
94default value to @code{nil} early in startup.
95@end defvar
96
b6954afd
RS
97@defun position-bytes position
98@tindex position-bytes
5ac343ac
RS
99Return the byte-position corresponding to buffer position
100@var{position} in the current buffer. This is 1 at the start of the
101buffer, and counts upward in bytes. If @var{position} is out of
102range, the value is @code{nil}.
b6954afd
RS
103@end defun
104
105@defun byte-to-position byte-position
106@tindex byte-to-position
107Return the buffer position corresponding to byte-position
35864124
LT
108@var{byte-position} in the current buffer. If @var{byte-position} is
109out of range, the value is @code{nil}.
b6954afd
RS
110@end defun
111
cc6d0d2c 112@defun multibyte-string-p string
b6954afd 113Return @code{t} if @var{string} is a multibyte string.
cc6d0d2c
RS
114@end defun
115
116@node Converting Representations
117@section Converting Text Representations
118
119 Emacs can convert unibyte text to multibyte; it can also convert
120multibyte text to unibyte, though this conversion loses information. In
121general these conversions happen when inserting text into a buffer, or
122when putting text from several strings together in one string. You can
123also explicitly convert a string's contents to either representation.
124
125 Emacs chooses the representation for a string based on the text that
126it is constructed from. The general rule is to convert unibyte text to
127multibyte text when combining it with other multibyte text, because the
128multibyte representation is more general and can hold whatever
129characters the unibyte text has.
130
131 When inserting text into a buffer, Emacs converts the text to the
132buffer's representation, as specified by
133@code{enable-multibyte-characters} in that buffer. In particular, when
134you insert multibyte text into a unibyte buffer, Emacs converts the text
135to unibyte, even though this conversion cannot in general preserve all
136the characters that might be in the multibyte text. The other natural
137alternative, to convert the buffer contents to multibyte, is not
138acceptable because the buffer's representation is a choice made by the
969fe9b5 139user that cannot be overridden automatically.
cc6d0d2c 140
ad800164 141 Converting unibyte text to multibyte text leaves @acronym{ASCII} characters
1e4d32f8 142unchanged, and likewise character codes 128 through 159. It converts
ad800164 143the non-@acronym{ASCII} codes 160 through 255 by adding the value
1e4d32f8
GM
144@code{nonascii-insert-offset} to each character code. By setting this
145variable, you specify which character set the unibyte characters
146correspond to (@pxref{Character Sets}). For example, if
147@code{nonascii-insert-offset} is 2048, which is @code{(- (make-char
ad800164 148'latin-iso8859-1) 128)}, then the unibyte non-@acronym{ASCII} characters
1e4d32f8
GM
149correspond to Latin 1. If it is 2688, which is @code{(- (make-char
150'greek-iso8859-7) 128)}, then they correspond to Greek letters.
cc6d0d2c 151
8241495d
RS
152 Converting multibyte text to unibyte is simpler: it discards all but
153the low 8 bits of each character code. If @code{nonascii-insert-offset}
154has a reasonable value, corresponding to the beginning of some character
155set, this conversion is the inverse of the other: converting unibyte
156text to multibyte and back to unibyte reproduces the original unibyte
157text.
cc6d0d2c 158
cc6d0d2c 159@defvar nonascii-insert-offset
ad800164 160This variable specifies the amount to add to a non-@acronym{ASCII} character
cc6d0d2c 161when converting unibyte text to multibyte. It also applies when
a9f0a989 162@code{self-insert-command} inserts a character in the unibyte
ad800164 163non-@acronym{ASCII} range, 128 through 255. However, the functions
7a063989 164@code{insert} and @code{insert-char} do not perform this conversion.
cc6d0d2c
RS
165
166The right value to use to select character set @var{cs} is @code{(-
a9f0a989 167(make-char @var{cs}) 128)}. If the value of
cc6d0d2c
RS
168@code{nonascii-insert-offset} is zero, then conversion actually uses the
169value for the Latin 1 character set, rather than zero.
170@end defvar
171
a9f0a989 172@defvar nonascii-translation-table
cc6d0d2c
RS
173This variable provides a more general alternative to
174@code{nonascii-insert-offset}. You can use it to specify independently
175how to translate each code in the range of 128 through 255 into a
7a063989 176multibyte character. The value should be a char-table, or @code{nil}.
969fe9b5 177If this is non-@code{nil}, it overrides @code{nonascii-insert-offset}.
cc6d0d2c
RS
178@end defvar
179
35864124
LT
180The next three functions either return the argument @var{string}, or a
181newly created string with no text properties.
182
cc6d0d2c
RS
183@defun string-make-unibyte string
184This function converts the text of @var{string} to unibyte
1911e6e5 185representation, if it isn't already, and returns the result. If
38eee91c
EZ
186@var{string} is a unibyte string, it is returned unchanged. Multibyte
187character codes are converted to unibyte according to
188@code{nonascii-translation-table} or, if that is @code{nil}, using
189@code{nonascii-insert-offset}. If the lookup in the translation table
190fails, this function takes just the low 8 bits of each character.
cc6d0d2c
RS
191@end defun
192
cc6d0d2c
RS
193@defun string-make-multibyte string
194This function converts the text of @var{string} to multibyte
1911e6e5 195representation, if it isn't already, and returns the result. If
35864124
LT
196@var{string} is a multibyte string or consists entirely of
197@acronym{ASCII} characters, it is returned unchanged. In particular,
198if @var{string} is unibyte and entirely @acronym{ASCII}, the returned
199string is unibyte. (When the characters are all @acronym{ASCII},
200Emacs primitives will treat the string the same way whether it is
201unibyte or multibyte.) If @var{string} is unibyte and contains
202non-@acronym{ASCII} characters, the function
203@code{unibyte-char-to-multibyte} is used to convert each unibyte
204character to a multibyte character.
cc6d0d2c
RS
205@end defun
206
131bf943
RS
207@defun string-to-multibyte string
208This function returns a multibyte string containing the same sequence
35864124
LT
209of character codes as @var{string}. Unlike
210@code{string-make-multibyte}, this function unconditionally returns a
211multibyte string. If @var{string} is a multibyte string, it is
212returned unchanged.
131bf943
RS
213@end defun
214
1ee89891
RS
215@defun multibyte-char-to-unibyte char
216This convert the multibyte character @var{char} to a unibyte
217character, based on @code{nonascii-translation-table} and
218@code{nonascii-insert-offset}.
219@end defun
220
221@defun unibyte-char-to-multibyte char
222This convert the unibyte character @var{char} to a multibyte
223character, based on @code{nonascii-translation-table} and
224@code{nonascii-insert-offset}.
225@end defun
226
cc6d0d2c
RS
227@node Selecting a Representation
228@section Selecting a Representation
229
230 Sometimes it is useful to examine an existing buffer or string as
231multibyte when it was unibyte, or vice versa.
232
cc6d0d2c
RS
233@defun set-buffer-multibyte multibyte
234Set the representation type of the current buffer. If @var{multibyte}
235is non-@code{nil}, the buffer becomes multibyte. If @var{multibyte}
236is @code{nil}, the buffer becomes unibyte.
237
238This function leaves the buffer contents unchanged when viewed as a
239sequence of bytes. As a consequence, it can change the contents viewed
240as characters; a sequence of two bytes which is treated as one character
241in multibyte representation will count as two characters in unibyte
7a063989
KH
242representation. Character codes 128 through 159 are an exception. They
243are represented by one byte in a unibyte buffer, but when the buffer is
244set to multibyte, they are converted to two-byte sequences, and vice
245versa.
cc6d0d2c
RS
246
247This function sets @code{enable-multibyte-characters} to record which
248representation is in use. It also adjusts various data in the buffer
969fe9b5
RS
249(including overlays, text properties and markers) so that they cover the
250same text as they did before.
b6954afd
RS
251
252You cannot use @code{set-buffer-multibyte} on an indirect buffer,
253because indirect buffers always inherit the representation of the
254base buffer.
cc6d0d2c
RS
255@end defun
256
cc6d0d2c
RS
257@defun string-as-unibyte string
258This function returns a string with the same bytes as @var{string} but
259treating each byte as a character. This means that the value may have
260more characters than @var{string} has.
261
b6954afd 262If @var{string} is already a unibyte string, then the value is
7f84d9ae
DL
263@var{string} itself. Otherwise it is a newly created string, with no
264text properties. If @var{string} is multibyte, any characters it
686ffe28 265contains of charset @code{eight-bit-control} or @code{eight-bit-graphic}
7f84d9ae 266are converted to the corresponding single byte.
cc6d0d2c
RS
267@end defun
268
cc6d0d2c
RS
269@defun string-as-multibyte string
270This function returns a string with the same bytes as @var{string} but
271treating each multibyte sequence as one character. This means that the
272value may have fewer characters than @var{string} has.
273
b6954afd 274If @var{string} is already a multibyte string, then the value is
7f84d9ae
DL
275@var{string} itself. Otherwise it is a newly created string, with no
276text properties. If @var{string} is unibyte and contains any individual
2778-bit bytes (i.e.@: not part of a multibyte form), they are converted to
686ffe28
RS
278the corresponding multibyte character of charset @code{eight-bit-control}
279or @code{eight-bit-graphic}.
cc6d0d2c
RS
280@end defun
281
282@node Character Codes
283@section Character Codes
284@cindex character codes
285
286 The unibyte and multibyte text representations use different character
287codes. The valid character codes for unibyte representation range from
2880 to 255---the values that can fit in one byte. The valid character
289codes for multibyte representation range from 0 to 524287, but not all
0ace421a 290values in that range are valid. The values 128 through 255 are not
1e4d32f8 291entirely proper in multibyte text, but they can occur if you do explicit
0ace421a 292encoding and decoding (@pxref{Explicit Encoding}). Some other character
ad800164 293codes cannot occur at all in multibyte text. Only the @acronym{ASCII} codes
1e4d32f8 2940 through 127 are completely legitimate in both representations.
cc6d0d2c 295
7a063989 296@defun char-valid-p charcode &optional genericp
0a58afcd
RS
297This returns @code{t} if @var{charcode} is valid (either for unibyte
298text or for multibyte text).
cc6d0d2c
RS
299
300@example
301(char-valid-p 65)
302 @result{} t
303(char-valid-p 256)
304 @result{} nil
305(char-valid-p 2248)
306 @result{} t
307@end example
7a063989 308
6fe50867 309If the optional argument @var{genericp} is non-@code{nil}, this
35864124
LT
310function also returns @code{t} if @var{charcode} is a generic
311character (@pxref{Splitting Characters}).
cc6d0d2c
RS
312@end defun
313
314@node Character Sets
315@section Character Sets
316@cindex character sets
317
318 Emacs classifies characters into various @dfn{character sets}, each of
319which has a name which is a symbol. Each character belongs to one and
320only one character set.
321
322 In general, there is one character set for each distinct script. For
323example, @code{latin-iso8859-1} is one character set,
324@code{greek-iso8859-7} is another, and @code{ascii} is another. An
969fe9b5
RS
325Emacs character set can hold at most 9025 characters; therefore, in some
326cases, characters that would logically be grouped together are split
a9f0a989
RS
327into several character sets. For example, one set of Chinese
328characters, generally known as Big 5, is divided into two Emacs
329character sets, @code{chinese-big5-1} and @code{chinese-big5-2}.
cc6d0d2c 330
ad800164
EZ
331 @acronym{ASCII} characters are in character set @code{ascii}. The
332non-@acronym{ASCII} characters 128 through 159 are in character set
4240c779
GM
333@code{eight-bit-control}, and codes 160 through 255 are in character set
334@code{eight-bit-graphic}.
335
cc6d0d2c 336@defun charsetp object
8241495d 337Returns @code{t} if @var{object} is a symbol that names a character set,
cc6d0d2c
RS
338@code{nil} otherwise.
339@end defun
340
35864124
LT
341@defvar charset-list
342The value is a list of all defined character set names.
343@end defvar
344
cc6d0d2c 345@defun charset-list
35864124
LT
346This function returns the value of @code{charset-list}. It is only
347provided for backward compatibility.
cc6d0d2c
RS
348@end defun
349
cc6d0d2c 350@defun char-charset character
b6954afd 351This function returns the name of the character set that @var{character}
35864124
LT
352belongs to, or the symbol @code{unknown} if @var{character} is not a
353valid character.
cc6d0d2c
RS
354@end defun
355
8241495d
RS
356@defun charset-plist charset
357@tindex charset-plist
358This function returns the charset property list of the character set
359@var{charset}. Although @var{charset} is a symbol, this is not the same
360as the property list of that symbol. Charset properties are used for
0f4da9ce 361special purposes within Emacs.
8241495d
RS
362@end defun
363
5ac343ac
RS
364@deffn Command list-charset-chars charset
365This command displays a list of characters in the character set
366@var{charset}.
367@end deffn
368
cc6d0d2c
RS
369@node Chars and Bytes
370@section Characters and Bytes
371@cindex bytes and characters
372
a9f0a989
RS
373@cindex introduction sequence
374@cindex dimension (of character set)
cc6d0d2c 375 In multibyte representation, each character occupies one or more
a9f0a989 376bytes. Each character set has an @dfn{introduction sequence}, which is
ad800164
EZ
377normally one or two bytes long. (Exception: the @code{ascii} character
378set and the @code{eight-bit-graphic} character set have a zero-length
7a063989
KH
379introduction sequence.) The introduction sequence is the beginning of
380the byte sequence for any character in the character set. The rest of
381the character's bytes distinguish it from the other characters in the
382same character set. Depending on the character set, there are either
383one or two distinguishing bytes; the number of such bytes is called the
384@dfn{dimension} of the character set.
a9f0a989
RS
385
386@defun charset-dimension charset
b6954afd
RS
387This function returns the dimension of @var{charset}; at present, the
388dimension is always 1 or 2.
389@end defun
390
391@defun charset-bytes charset
392@tindex charset-bytes
393This function returns the number of bytes used to represent a character
394in character set @var{charset}.
a9f0a989
RS
395@end defun
396
397 This is the simplest way to determine the byte length of a character
398set's introduction sequence:
399
400@example
b6954afd 401(- (charset-bytes @var{charset})
a9f0a989
RS
402 (charset-dimension @var{charset}))
403@end example
404
405@node Splitting Characters
406@section Splitting Characters
407
408 The functions in this section convert between characters and the byte
409values used to represent them. For most purposes, there is no need to
410be concerned with the sequence of bytes used to represent a character,
969fe9b5 411because Emacs translates automatically when necessary.
cc6d0d2c 412
cc6d0d2c
RS
413@defun split-char character
414Return a list containing the name of the character set of
a9f0a989
RS
415@var{character}, followed by one or two byte values (integers) which
416identify @var{character} within that character set. The number of byte
417values is the character set's dimension.
cc6d0d2c 418
35864124
LT
419If @var{character} is invalid as a character code, @code{split-char}
420returns a list consisting of the symbol @code{unknown} and @var{character}.
421
cc6d0d2c
RS
422@example
423(split-char 2248)
424 @result{} (latin-iso8859-1 72)
425(split-char 65)
426 @result{} (ascii 65)
7a063989
KH
427(split-char 128)
428 @result{} (eight-bit-control 128)
cc6d0d2c
RS
429@end example
430@end defun
431
e8262f40
DL
432@defun make-char charset &optional code1 code2
433This function returns the character in character set @var{charset} whose
434position codes are @var{code1} and @var{code2}. This is roughly the
435inverse of @code{split-char}. Normally, you should specify either one
436or both of @var{code1} and @var{code2} according to the dimension of
437@var{charset}. For example,
cc6d0d2c
RS
438
439@example
440(make-char 'latin-iso8859-1 72)
441 @result{} 2248
442@end example
0f4da9ce
DL
443
444Actually, the eighth bit of both @var{code1} and @var{code2} is zeroed
445before they are used to index @var{charset}. Thus you may use, for
446instance, an ISO 8859 character code rather than subtracting 128, as
447is necessary to index the corresponding Emacs charset.
cc6d0d2c
RS
448@end defun
449
a9f0a989
RS
450@cindex generic characters
451 If you call @code{make-char} with no @var{byte-values}, the result is
452a @dfn{generic character} which stands for @var{charset}. A generic
453character is an integer, but it is @emph{not} valid for insertion in the
454buffer as a character. It can be used in @code{char-table-range} to
455refer to the whole character set (@pxref{Char-Tables}).
456@code{char-valid-p} returns @code{nil} for generic characters.
457For example:
458
459@example
460(make-char 'latin-iso8859-1)
461 @result{} 2176
462(char-valid-p 2176)
463 @result{} nil
7a063989
KH
464(char-valid-p 2176 t)
465 @result{} t
a9f0a989
RS
466(split-char 2176)
467 @result{} (latin-iso8859-1 0)
468@end example
469
ad800164
EZ
470The character sets @code{ascii}, @code{eight-bit-control}, and
471@code{eight-bit-graphic} don't have corresponding generic characters. If
e8262f40
DL
472@var{charset} is one of them and you don't supply @var{code1},
473@code{make-char} returns the character code corresponding to the
474smallest code in @var{charset}.
7a063989 475
a9f0a989
RS
476@node Scanning Charsets
477@section Scanning for Character Sets
478
479 Sometimes it is useful to find out which character sets appear in a
480part of a buffer or a string. One use for this is in determining which
481coding systems (@pxref{Coding Systems}) are capable of representing all
482of the text in question.
483
5ac343ac
RS
484@defun charset-after &optional pos
485This function return the charset of a character in the current buffer
486at position @var{pos}. If @var{pos} is omitted or @code{nil}, it
5a36d834 487defaults to the current value of point. If @var{pos} is out of range,
5ac343ac
RS
488the value is @code{nil}.
489@end defun
490
a9f0a989 491@defun find-charset-region beg end &optional translation
a9f0a989
RS
492This function returns a list of the character sets that appear in the
493current buffer between positions @var{beg} and @var{end}.
494
495The optional argument @var{translation} specifies a translation table to
496be used in scanning the text (@pxref{Translation of Characters}). If it
497is non-@code{nil}, then each character in the region is translated
498through this table, and the value returned describes the translated
499characters instead of the characters actually in the buffer.
a265079f 500@end defun
a9f0a989
RS
501
502@defun find-charset-string string &optional translation
b6954afd
RS
503This function returns a list of the character sets that appear in the
504string @var{string}. It is just like @code{find-charset-region}, except
505that it applies to the contents of @var{string} instead of part of the
506current buffer.
a9f0a989
RS
507@end defun
508
509@node Translation of Characters
510@section Translation of Characters
511@cindex character translation tables
512@cindex translation tables
513
35864124
LT
514 A @dfn{translation table} is a char-table that specifies a mapping
515of characters into characters. These tables are used in encoding and
516decoding, and for other purposes. Some coding systems specify their
517own particular translation tables; there are also default translation
518tables which apply to all other coding systems.
a9f0a989 519
a3d3f60d
RS
520 For instance, the coding-system @code{utf-8} has a translation table
521that maps characters of various charsets (e.g.,
522@code{latin-iso8859-@var{x}}) into Unicode character sets. This way,
523it can encode Latin-2 characters into UTF-8. Meanwhile,
524@code{unify-8859-on-decoding-mode} operates by specifying
525@code{standard-translation-table-for-decode} to translate
526Latin-@var{x} characters into corresponding Unicode characters.
527
8241495d
RS
528@defun make-translation-table &rest translations
529This function returns a translation table based on the argument
f57b6e64
DL
530@var{translations}. Each element of @var{translations} should be a
531list of elements of the form @code{(@var{from} . @var{to})}; this says
532to translate the character @var{from} into @var{to}.
a9f0a989 533
c04c052b
DL
534The arguments and the forms in each argument are processed in order,
535and if a previous form already translates @var{to} to some other
536character, say @var{to-alt}, @var{from} is also translated to
537@var{to-alt}.
538
a9f0a989
RS
539You can also map one whole character set into another character set with
540the same dimension. To do this, you specify a generic character (which
541designates a character set) for @var{from} (@pxref{Splitting Characters}).
35864124
LT
542In this case, if @var{to} is also a generic character, its character
543set should have the same dimension as @var{from}'s. Then the
544translation table translates each character of @var{from}'s character
545set into the corresponding character of @var{to}'s character set. If
546@var{from} is a generic character and @var{to} is an ordinary
547character, then the translation table translates every character of
548@var{from}'s character set into @var{to}.
a9f0a989
RS
549@end defun
550
551 In decoding, the translation table's translations are applied to the
552characters that result from ordinary decoding. If a coding system has
35864124
LT
553property @code{translation-table-for-decode}, that specifies the
554translation table to use. (This is a property of the coding system,
555as returned by @code{coding-system-get}, not a property of the symbol
556that is the coding system's name. @xref{Coding System Basics,, Basic
557Concepts of Coding Systems}.) Otherwise, if
558@code{standard-translation-table-for-decode} is non-@code{nil},
559decoding uses that table.
a9f0a989
RS
560
561 In encoding, the translation table's translations are applied to the
562characters in the buffer, and the result of translation is actually
563encoded. If a coding system has property
35864124
LT
564@code{translation-table-for-encode}, that specifies the translation
565table to use. Otherwise the variable
b1f687a2
RS
566@code{standard-translation-table-for-encode} specifies the translation
567table.
a9f0a989 568
b1f687a2 569@defvar standard-translation-table-for-decode
a9f0a989
RS
570This is the default translation table for decoding, for
571coding systems that don't specify any other translation table.
572@end defvar
573
b1f687a2 574@defvar standard-translation-table-for-encode
a9f0a989
RS
575This is the default translation table for encoding, for
576coding systems that don't specify any other translation table.
577@end defvar
578
131bf943
RS
579@defvar translation-table-for-input
580Self-inserting characters are translated through this translation
fa27fb28
RS
581table before they are inserted. Search commands also translate their
582input through this table, so they can compare more reliably with
583what's in the buffer.
a3d3f60d
RS
584
585@code{set-buffer-file-coding-system} sets this variable so that your
586keyboard input gets translated into the character sets that the buffer
fa27fb28
RS
587is likely to contain. This variable automatically becomes
588buffer-local when set.
131bf943
RS
589@end defvar
590
cc6d0d2c
RS
591@node Coding Systems
592@section Coding Systems
593
594@cindex coding system
595 When Emacs reads or writes a file, and when Emacs sends text to a
596subprocess or receives text from a subprocess, it normally performs
597character code conversion and end-of-line conversion as specified
598by a particular @dfn{coding system}.
599
8241495d
RS
600 How to define a coding system is an arcane matter, and is not
601documented here.
b6954afd 602
a9f0a989 603@menu
5557b83b
RS
604* Coding System Basics:: Basic concepts.
605* Encoding and I/O:: How file I/O functions handle coding systems.
606* Lisp and Coding Systems:: Functions to operate on coding system names.
607* User-Chosen Coding Systems:: Asking the user to choose a coding system.
608* Default Coding Systems:: Controlling the default choices.
609* Specifying Coding Systems:: Requesting a particular coding system
610 for a single file operation.
611* Explicit Encoding:: Encoding or decoding text without doing I/O.
612* Terminal I/O Encoding:: Use of encoding for terminal I/O.
613* MS-DOS File Types:: How DOS "text" and "binary" files
614 relate to coding systems.
a9f0a989
RS
615@end menu
616
617@node Coding System Basics
618@subsection Basic Concepts of Coding Systems
619
cc6d0d2c
RS
620@cindex character code conversion
621 @dfn{Character code conversion} involves conversion between the encoding
622used inside Emacs and some other encoding. Emacs supports many
623different encodings, in that it can convert to and from them. For
624example, it can convert text to or from encodings such as Latin 1, Latin
6252, Latin 3, Latin 4, Latin 5, and several variants of ISO 2022. In some
626cases, Emacs supports several alternative encodings for the same
627characters; for example, there are three coding systems for the Cyrillic
628(Russian) alphabet: ISO, Alternativnyj, and KOI8.
629
cc6d0d2c 630 Most coding systems specify a particular character code for
8241495d
RS
631conversion, but some of them leave the choice unspecified---to be chosen
632heuristically for each file, based on the data.
cc6d0d2c 633
aa945b59
RS
634 In general, a coding system doesn't guarantee roundtrip identity:
635decoding a byte sequence using coding system, then encoding the
636resulting text in the same coding system, can produce a different byte
637sequence. However, the following coding systems do guarantee that the
638byte sequence will be the same as what you originally decoded:
6fa88620
KH
639
640@quotation
641chinese-big5 chinese-iso-8bit cyrillic-iso-8bit emacs-mule
642greek-iso-8bit hebrew-iso-8bit iso-latin-1 iso-latin-2 iso-latin-3
643iso-latin-4 iso-latin-5 iso-latin-8 iso-latin-9 iso-safe
644japanese-iso-8bit japanese-shift-jis korean-iso-8bit raw-text
645@end quotation
646
aa945b59
RS
647 Encoding buffer text and then decoding the result can also fail to
648reproduce the original text. For instance, if you encode Latin-2
8b918214
RS
649characters with @code{utf-8} and decode the result using the same
650coding system, you'll get Unicode characters (of charset
aa945b59
RS
651@code{mule-unicode-0100-24ff}). If you encode Unicode characters with
652@code{iso-latin-2} and decode the result with the same coding system,
653you'll get Latin-2 characters.
6fa88620 654
969fe9b5
RS
655@cindex end of line conversion
656 @dfn{End of line conversion} handles three different conventions used
657on various systems for representing end of line in files. The Unix
658convention is to use the linefeed character (also called newline). The
8241495d
RS
659DOS convention is to use a carriage-return and a linefeed at the end of
660a line. The Mac convention is to use just carriage-return.
969fe9b5 661
cc6d0d2c
RS
662@cindex base coding system
663@cindex variant coding system
664 @dfn{Base coding systems} such as @code{latin-1} leave the end-of-line
665conversion unspecified, to be chosen based on the data. @dfn{Variant
666coding systems} such as @code{latin-1-unix}, @code{latin-1-dos} and
667@code{latin-1-mac} specify the end-of-line conversion explicitly as
a9f0a989 668well. Most base coding systems have three corresponding variants whose
cc6d0d2c
RS
669names are formed by adding @samp{-unix}, @samp{-dos} and @samp{-mac}.
670
a9f0a989
RS
671 The coding system @code{raw-text} is special in that it prevents
672character code conversion, and causes the buffer visited with that
673coding system to be a unibyte buffer. It does not specify the
674end-of-line conversion, allowing that to be determined as usual by the
675data, and has the usual three variants which specify the end-of-line
676conversion. @code{no-conversion} is equivalent to @code{raw-text-unix}:
677it specifies no conversion of either character codes or end-of-line.
678
679 The coding system @code{emacs-mule} specifies that the data is
680represented in the internal Emacs encoding. This is like
681@code{raw-text} in that no code conversion happens, but different in
682that the result is multibyte data.
683
684@defun coding-system-get coding-system property
a9f0a989
RS
685This function returns the specified property of the coding system
686@var{coding-system}. Most coding system properties exist for internal
687purposes, but one that you might find useful is @code{mime-charset}.
688That property's value is the name used in MIME for the character coding
689which this coding system can read and write. Examples:
690
691@example
692(coding-system-get 'iso-latin-1 'mime-charset)
693 @result{} iso-8859-1
694(coding-system-get 'iso-2022-cn 'mime-charset)
695 @result{} iso-2022-cn
696(coding-system-get 'cyrillic-koi8 'mime-charset)
697 @result{} koi8-r
698@end example
699
700The value of the @code{mime-charset} property is also defined
701as an alias for the coding system.
702@end defun
703
704@node Encoding and I/O
705@subsection Encoding and I/O
706
1911e6e5 707 The principal purpose of coding systems is for use in reading and
a9f0a989
RS
708writing files. The function @code{insert-file-contents} uses
709a coding system for decoding the file data, and @code{write-region}
710uses one to encode the buffer contents.
711
712 You can specify the coding system to use either explicitly
5ac343ac 713(@pxref{Specifying Coding Systems}), or implicitly using a default
a9f0a989
RS
714mechanism (@pxref{Default Coding Systems}). But these methods may not
715completely specify what to do. For example, they may choose a coding
716system such as @code{undefined} which leaves the character code
717conversion to be determined from the data. In these cases, the I/O
718operation finishes the job of choosing a coding system. Very often
719you will want to find out afterwards which coding system was chosen.
720
721@defvar buffer-file-coding-system
475aab0d
CY
722This buffer-local variable records the coding system that was used to visit
723the current buffer. It is used for saving the buffer, and for writing part
1b02d12c
EZ
724of the buffer with @code{write-region}. If the text to be written
725cannot be safely encoded using the coding system specified by this
726variable, these operations select an alternative encoding by calling
727the function @code{select-safe-coding-system} (@pxref{User-Chosen
728Coding Systems}). If selecting a different encoding requires to ask
729the user to specify a coding system, @code{buffer-file-coding-system}
730is updated to the newly selected coding system.
731
732@code{buffer-file-coding-system} does @emph{not} affect sending text
b6954afd 733to a subprocess.
a9f0a989
RS
734@end defvar
735
736@defvar save-buffer-coding-system
7a063989
KH
737This variable specifies the coding system for saving the buffer (by
738overriding @code{buffer-file-coding-system}). Note that it is not used
739for @code{write-region}.
8241495d
RS
740
741When a command to save the buffer starts out to use
7a063989
KH
742@code{buffer-file-coding-system} (or @code{save-buffer-coding-system}),
743and that coding system cannot handle
8241495d 744the actual text in the buffer, the command asks the user to choose
1b02d12c
EZ
745another coding system (by calling @code{select-safe-coding-system}).
746After that happens, the command also updates
747@code{buffer-file-coding-system} to represent the coding system that
748the user specified.
a9f0a989
RS
749@end defvar
750
751@defvar last-coding-system-used
a9f0a989
RS
752I/O operations for files and subprocesses set this variable to the
753coding system name that was used. The explicit encoding and decoding
754functions (@pxref{Explicit Encoding}) set it too.
755
756@strong{Warning:} Since receiving subprocess output sets this variable,
8241495d
RS
757it can change whenever Emacs waits; therefore, you should copy the
758value shortly after the function call that stores the value you are
a9f0a989
RS
759interested in.
760@end defvar
761
2eb4136f
RS
762 The variable @code{selection-coding-system} specifies how to encode
763selections for the window system. @xref{Window System Selections}.
764
1ee89891
RS
765@defvar file-name-coding-system
766The variable @code{file-name-coding-system} specifies the coding
767system to use for encoding file names. Emacs encodes file names using
768that coding system for all file operations. If
769@code{file-name-coding-system} is @code{nil}, Emacs uses a default
770coding system determined by the selected language environment. In the
771default language environment, any non-@acronym{ASCII} characters in
772file names are not encoded specially; they appear in the file system
773using the internal Emacs representation.
774@end defvar
775
776 @strong{Warning:} if you change @code{file-name-coding-system} (or
777the language environment) in the middle of an Emacs session, problems
778can result if you have already visited files whose names were encoded
779using the earlier coding system and are handled differently under the
780new coding system. If you try to save one of these buffers under the
781visited file name, saving may use the wrong file name, or it may get
782an error. If such a problem happens, use @kbd{C-x C-w} to specify a
783new file name for that buffer.
784
969fe9b5
RS
785@node Lisp and Coding Systems
786@subsection Coding Systems in Lisp
787
8241495d 788 Here are the Lisp facilities for working with coding systems:
cc6d0d2c 789
cc6d0d2c
RS
790@defun coding-system-list &optional base-only
791This function returns a list of all coding system names (symbols). If
792@var{base-only} is non-@code{nil}, the value includes only the
7a063989
KH
793base coding systems. Otherwise, it includes alias and variant coding
794systems as well.
cc6d0d2c
RS
795@end defun
796
cc6d0d2c
RS
797@defun coding-system-p object
798This function returns @code{t} if @var{object} is a coding system
35864124 799name or @code{nil}.
cc6d0d2c
RS
800@end defun
801
cc6d0d2c
RS
802@defun check-coding-system coding-system
803This function checks the validity of @var{coding-system}.
804If that is valid, it returns @var{coding-system}.
805Otherwise it signals an error with condition @code{coding-system-error}.
806@end defun
807
a9f0a989 808@defun coding-system-change-eol-conversion coding-system eol-type
a9f0a989 809This function returns a coding system which is like @var{coding-system}
1911e6e5 810except for its eol conversion, which is specified by @code{eol-type}.
a9f0a989
RS
811@var{eol-type} should be @code{unix}, @code{dos}, @code{mac}, or
812@code{nil}. If it is @code{nil}, the returned coding system determines
813the end-of-line conversion from the data.
35864124
LT
814
815@var{eol-type} may also be 0, 1 or 2, standing for @code{unix},
070b546b 816@code{dos} and @code{mac}, respectively.
a9f0a989 817@end defun
969fe9b5 818
a9f0a989 819@defun coding-system-change-text-conversion eol-coding text-coding
a9f0a989
RS
820This function returns a coding system which uses the end-of-line
821conversion of @var{eol-coding}, and the text conversion of
822@var{text-coding}. If @var{text-coding} is @code{nil}, it returns
823@code{undecided}, or one of its variants according to @var{eol-coding}.
969fe9b5
RS
824@end defun
825
a9f0a989 826@defun find-coding-systems-region from to
a9f0a989
RS
827This function returns a list of coding systems that could be used to
828encode a text between @var{from} and @var{to}. All coding systems in
829the list can safely encode any multibyte characters in that portion of
830the text.
831
832If the text contains no multibyte characters, the function returns the
833list @code{(undecided)}.
834@end defun
835
836@defun find-coding-systems-string string
a9f0a989
RS
837This function returns a list of coding systems that could be used to
838encode the text of @var{string}. All coding systems in the list can
839safely encode any multibyte characters in @var{string}. If the text
840contains no multibyte characters, this returns the list
841@code{(undecided)}.
842@end defun
843
844@defun find-coding-systems-for-charsets charsets
a9f0a989
RS
845This function returns a list of coding systems that could be used to
846encode all the character sets in the list @var{charsets}.
847@end defun
848
849@defun detect-coding-region start end &optional highest
cc6d0d2c 850This function chooses a plausible coding system for decoding the text
0ace421a 851from @var{start} to @var{end}. This text should be a byte sequence
969fe9b5 852(@pxref{Explicit Encoding}).
cc6d0d2c 853
a9f0a989 854Normally this function returns a list of coding systems that could
cc6d0d2c 855handle decoding the text that was scanned. They are listed in order of
a9f0a989
RS
856decreasing priority. But if @var{highest} is non-@code{nil}, then the
857return value is just one coding system, the one that is highest in
858priority.
859
ad800164 860If the region contains only @acronym{ASCII} characters, the value
35864124
LT
861is @code{undecided} or @code{(undecided)}, or a variant specifying
862end-of-line conversion, if that can be deduced from the text.
cc6d0d2c
RS
863@end defun
864
35864124 865@defun detect-coding-string string &optional highest
cc6d0d2c
RS
866This function is like @code{detect-coding-region} except that it
867operates on the contents of @var{string} instead of bytes in the buffer.
1911e6e5
RS
868@end defun
869
35864124
LT
870 @xref{Coding systems for a subprocess,, Process Information}, in
871particular the description of the functions
872@code{process-coding-system} and @code{set-process-coding-system}, for
873how to examine or set the coding systems used for I/O to a subprocess.
1911e6e5
RS
874
875@node User-Chosen Coding Systems
876@subsection User-Chosen Coding Systems
877
1b02d12c 878@cindex select safe coding system
35864124 879@defun select-safe-coding-system from to &optional default-coding-system accept-default-p file
bf23b477
EZ
880This function selects a coding system for encoding specified text,
881asking the user to choose if necessary. Normally the specified text
35864124
LT
882is the text in the current buffer between @var{from} and @var{to}. If
883@var{from} is a string, the string specifies the text to encode, and
884@var{to} is ignored.
bf23b477
EZ
885
886If @var{default-coding-system} is non-@code{nil}, that is the first
887coding system to try; if that can handle the text,
888@code{select-safe-coding-system} returns that coding system. It can
889also be a list of coding systems; then the function tries each of them
35864124
LT
890one by one. After trying all of them, it next tries the current
891buffer's value of @code{buffer-file-coding-system} (if it is not
892@code{undecided}), then the value of
893@code{default-buffer-file-coding-system} and finally the user's most
894preferred coding system, which the user can set using the command
895@code{prefer-coding-system} (@pxref{Recognize Coding,, Recognizing
896Coding Systems, emacs, The GNU Emacs Manual}).
bf23b477
EZ
897
898If one of those coding systems can safely encode all the specified
899text, @code{select-safe-coding-system} chooses it and returns it.
900Otherwise, it asks the user to choose from a list of coding systems
901which can encode all the text, and returns the user's choice.
902
35864124
LT
903@var{default-coding-system} can also be a list whose first element is
904t and whose other elements are coding systems. Then, if no coding
905system in the list can handle the text, @code{select-safe-coding-system}
906queries the user immediately, without trying any of the three
907alternatives described above.
908
bf23b477 909The optional argument @var{accept-default-p}, if non-@code{nil},
35864124
LT
910should be a function to determine whether a coding system selected
911without user interaction is acceptable. @code{select-safe-coding-system}
912calls this function with one argument, the base coding system of the
913selected coding system. If @var{accept-default-p} returns @code{nil},
914@code{select-safe-coding-system} rejects the silently selected coding
915system, and asks the user to select a coding system from a list of
916possible candidates.
bf23b477
EZ
917
918@vindex select-safe-coding-system-accept-default-p
919If the variable @code{select-safe-coding-system-accept-default-p} is
920non-@code{nil}, its value overrides the value of
921@var{accept-default-p}.
35864124
LT
922
923As a final step, before returning the chosen coding system,
924@code{select-safe-coding-system} checks whether that coding system is
925consistent with what would be selected if the contents of the region
926were read from a file. (If not, this could lead to data corruption in
927a file subsequently re-visited and edited.) Normally,
928@code{select-safe-coding-system} uses @code{buffer-file-name} as the
929file for this purpose, but if @var{file} is non-@code{nil}, it uses
930that file instead (this can be relevant for @code{write-region} and
931similar functions). If it detects an apparent inconsistency,
932@code{select-safe-coding-system} queries the user before selecting the
933coding system.
969fe9b5
RS
934@end defun
935
936 Here are two functions you can use to let the user specify a coding
937system, with completion. @xref{Completion}.
938
a9f0a989 939@defun read-coding-system prompt &optional default
969fe9b5
RS
940This function reads a coding system using the minibuffer, prompting with
941string @var{prompt}, and returns the coding system name as a symbol. If
942the user enters null input, @var{default} specifies which coding system
943to return. It should be a symbol or a string.
944@end defun
945
969fe9b5
RS
946@defun read-non-nil-coding-system prompt
947This function reads a coding system using the minibuffer, prompting with
a9f0a989 948string @var{prompt}, and returns the coding system name as a symbol. If
969fe9b5
RS
949the user tries to enter null input, it asks the user to try again.
950@xref{Coding Systems}.
cc6d0d2c
RS
951@end defun
952
953@node Default Coding Systems
a9f0a989 954@subsection Default Coding Systems
cc6d0d2c 955
a9f0a989
RS
956 This section describes variables that specify the default coding
957system for certain files or when running certain subprograms, and the
1911e6e5 958function that I/O operations use to access them.
a9f0a989
RS
959
960 The idea of these variables is that you set them once and for all to the
961defaults you want, and then do not change them again. To specify a
962particular coding system for a particular operation in a Lisp program,
963don't change these variables; instead, override them using
964@code{coding-system-for-read} and @code{coding-system-for-write}
965(@pxref{Specifying Coding Systems}).
cc6d0d2c 966
bf23b477
EZ
967@defvar auto-coding-regexp-alist
968This variable is an alist of text patterns and corresponding coding
969systems. Each element has the form @code{(@var{regexp}
970. @var{coding-system})}; a file whose first few kilobytes match
971@var{regexp} is decoded with @var{coding-system} when its contents are
972read into a buffer. The settings in this alist take priority over
973@code{coding:} tags in the files and the contents of
974@code{file-coding-system-alist} (see below). The default value is set
975so that Emacs automatically recognizes mail files in Babyl format and
976reads them with no code conversions.
977@end defvar
978
cc6d0d2c
RS
979@defvar file-coding-system-alist
980This variable is an alist that specifies the coding systems to use for
981reading and writing particular files. Each element has the form
982@code{(@var{pattern} . @var{coding})}, where @var{pattern} is a regular
983expression that matches certain file names. The element applies to file
984names that match @var{pattern}.
985
35864124 986The @sc{cdr} of the element, @var{coding}, should be either a coding
8241495d
RS
987system, a cons cell containing two coding systems, or a function name (a
988symbol with a function definition). If @var{coding} is a coding system,
989that coding system is used for both reading the file and writing it. If
35864124
LT
990@var{coding} is a cons cell containing two coding systems, its @sc{car}
991specifies the coding system for decoding, and its @sc{cdr} specifies the
8241495d
RS
992coding system for encoding.
993
35864124
LT
994If @var{coding} is a function name, the function should take one
995argument, a list of all arguments passed to
996@code{find-operation-coding-system}. It must return a coding system
997or a cons cell containing two coding systems. This value has the same
998meaning as described above.
cc6d0d2c
RS
999@end defvar
1000
cc6d0d2c
RS
1001@defvar process-coding-system-alist
1002This variable is an alist specifying which coding systems to use for a
1003subprocess, depending on which program is running in the subprocess. It
1004works like @code{file-coding-system-alist}, except that @var{pattern} is
1005matched against the program name used to start the subprocess. The coding
1006system or systems specified in this alist are used to initialize the
1007coding systems used for I/O to the subprocess, but you can specify
1008other coding systems later using @code{set-process-coding-system}.
1009@end defvar
1010
8241495d
RS
1011 @strong{Warning:} Coding systems such as @code{undecided}, which
1012determine the coding system from the data, do not work entirely reliably
1911e6e5 1013with asynchronous subprocess output. This is because Emacs handles
a9f0a989
RS
1014asynchronous subprocess output in batches, as it arrives. If the coding
1015system leaves the character code conversion unspecified, or leaves the
1016end-of-line conversion unspecified, Emacs must try to detect the proper
1017conversion from one batch at a time, and this does not always work.
1018
1019 Therefore, with an asynchronous subprocess, if at all possible, use a
1020coding system which determines both the character code conversion and
1021the end of line conversion---that is, one like @code{latin-1-unix},
1022rather than @code{undecided} or @code{latin-1}.
1023
cc6d0d2c
RS
1024@defvar network-coding-system-alist
1025This variable is an alist that specifies the coding system to use for
1026network streams. It works much like @code{file-coding-system-alist},
969fe9b5 1027with the difference that the @var{pattern} in an element may be either a
cc6d0d2c
RS
1028port number or a regular expression. If it is a regular expression, it
1029is matched against the network service name used to open the network
1030stream.
1031@end defvar
1032
cc6d0d2c
RS
1033@defvar default-process-coding-system
1034This variable specifies the coding systems to use for subprocess (and
1035network stream) input and output, when nothing else specifies what to
1036do.
1037
a9f0a989
RS
1038The value should be a cons cell of the form @code{(@var{input-coding}
1039. @var{output-coding})}. Here @var{input-coding} applies to input from
1040the subprocess, and @var{output-coding} applies to output to it.
cc6d0d2c
RS
1041@end defvar
1042
131bf943
RS
1043@defvar auto-coding-functions
1044This variable holds a list of functions that try to determine a
1045coding system for a file based on its undecoded contents.
1046
1047Each function in this list should be written to look at text in the
1048current buffer, but should not modify it in any way. The buffer will
1049contain undecoded text of parts of the file. Each function should
1050take one argument, @var{size}, which tells it how many characters to
1051look at, starting from point. If the function succeeds in determining
1052a coding system for the file, it should return that coding system.
1053Otherwise, it should return @code{nil}.
1054
1055If a file has a @samp{coding:} tag, that takes precedence, so these
1056functions won't be called.
1057@end defvar
1058
a9f0a989 1059@defun find-operation-coding-system operation &rest arguments
a9f0a989
RS
1060This function returns the coding system to use (by default) for
1061performing @var{operation} with @var{arguments}. The value has this
1062form:
1063
1064@example
35864124 1065(@var{decoding-system} . @var{encoding-system})
a9f0a989
RS
1066@end example
1067
1068The first element, @var{decoding-system}, is the coding system to use
1069for decoding (in case @var{operation} does decoding), and
1070@var{encoding-system} is the coding system for encoding (in case
1071@var{operation} does encoding).
1072
342fd6cd
RS
1073The argument @var{operation} should be a symbol, any one of
1074@code{insert-file-contents}, @code{write-region},
1075@code{start-process}, @code{call-process}, @code{call-process-region},
1076or @code{open-network-stream}. These are the names of the Emacs I/O
1077primitives that can do coding system conversion.
a9f0a989
RS
1078
1079The remaining arguments should be the same arguments that might be given
8241495d 1080to that I/O primitive. Depending on the primitive, one of those
a9f0a989
RS
1081arguments is selected as the @dfn{target}. For example, if
1082@var{operation} does file I/O, whichever argument specifies the file
1083name is the target. For subprocess primitives, the process name is the
1084target. For @code{open-network-stream}, the target is the service name
1085or port number.
1086
342fd6cd
RS
1087Depending on @var{operation}, this function looks up the target in
1088@code{file-coding-system-alist}, @code{process-coding-system-alist},
1089or @code{network-coding-system-alist}.
a9f0a989
RS
1090@end defun
1091
cc6d0d2c 1092@node Specifying Coding Systems
a9f0a989 1093@subsection Specifying a Coding System for One Operation
cc6d0d2c
RS
1094
1095 You can specify the coding system for a specific operation by binding
1096the variables @code{coding-system-for-read} and/or
1097@code{coding-system-for-write}.
1098
cc6d0d2c
RS
1099@defvar coding-system-for-read
1100If this variable is non-@code{nil}, it specifies the coding system to
1101use for reading a file, or for input from a synchronous subprocess.
1102
1103It also applies to any asynchronous subprocess or network stream, but in
1104a different way: the value of @code{coding-system-for-read} when you
1105start the subprocess or open the network stream specifies the input
1106decoding method for that subprocess or network stream. It remains in
1107use for that subprocess or network stream unless and until overridden.
1108
1109The right way to use this variable is to bind it with @code{let} for a
1110specific I/O operation. Its global value is normally @code{nil}, and
1111you should not globally set it to any other value. Here is an example
1112of the right way to use the variable:
1113
1114@example
1115;; @r{Read the file with no character code conversion.}
ad800164 1116;; @r{Assume @acronym{crlf} represents end-of-line.}
a3d3f60d 1117(let ((coding-system-for-read 'emacs-mule-dos))
cc6d0d2c
RS
1118 (insert-file-contents filename))
1119@end example
1120
1121When its value is non-@code{nil}, @code{coding-system-for-read} takes
a9f0a989 1122precedence over all other methods of specifying a coding system to use for
cc6d0d2c
RS
1123input, including @code{file-coding-system-alist},
1124@code{process-coding-system-alist} and
1125@code{network-coding-system-alist}.
1126@end defvar
1127
cc6d0d2c
RS
1128@defvar coding-system-for-write
1129This works much like @code{coding-system-for-read}, except that it
1130applies to output rather than input. It affects writing to files,
b6954afd 1131as well as sending output to subprocesses and net connections.
cc6d0d2c
RS
1132
1133When a single operation does both input and output, as do
1134@code{call-process-region} and @code{start-process}, both
1135@code{coding-system-for-read} and @code{coding-system-for-write}
1136affect it.
1137@end defvar
1138
cc6d0d2c
RS
1139@defvar inhibit-eol-conversion
1140When this variable is non-@code{nil}, no end-of-line conversion is done,
1141no matter which coding system is specified. This applies to all the
1142Emacs I/O and subprocess primitives, and to the explicit encoding and
1143decoding functions (@pxref{Explicit Encoding}).
1144@end defvar
1145
cc6d0d2c 1146@node Explicit Encoding
a9f0a989 1147@subsection Explicit Encoding and Decoding
cc6d0d2c
RS
1148@cindex encoding text
1149@cindex decoding text
1150
1151 All the operations that transfer text in and out of Emacs have the
1152ability to use a coding system to encode or decode the text.
1153You can also explicitly encode and decode text using the functions
1154in this section.
1155
cc6d0d2c 1156 The result of encoding, and the input to decoding, are not ordinary
0ace421a
GM
1157text. They logically consist of a series of byte values; that is, a
1158series of characters whose codes are in the range 0 through 255. In a
1159multibyte buffer or string, character codes 128 through 159 are
1160represented by multibyte sequences, but this is invisible to Lisp
1161programs.
1162
1163 The usual way to read a file into a buffer as a sequence of bytes, so
1164you can decode the contents explicitly, is with
1165@code{insert-file-contents-literally} (@pxref{Reading from Files});
1166alternatively, specify a non-@code{nil} @var{rawfile} argument when
1167visiting a file with @code{find-file-noselect}. These methods result in
1168a unibyte buffer.
1169
1170 The usual way to use the byte sequence that results from explicitly
1171encoding text is to copy it to a file or process---for example, to write
1172it with @code{write-region} (@pxref{Writing to Files}), and suppress
1173encoding by binding @code{coding-system-for-write} to
1174@code{no-conversion}.
b6954afd
RS
1175
1176 Here are the functions to perform explicit encoding or decoding. The
0ace421a
GM
1177decoding functions produce sequences of bytes; the encoding functions
1178are meant to operate on sequences of bytes. All of these functions
1179discard text properties.
1911e6e5 1180
35864124
LT
1181@deffn Command encode-coding-region start end coding-system
1182This command encodes the text from @var{start} to @var{end} according
969fe9b5 1183to coding system @var{coding-system}. The encoded text replaces the
0ace421a
GM
1184original text in the buffer. The result of encoding is logically a
1185sequence of bytes, but the buffer remains multibyte if it was multibyte
1186before.
cc6d0d2c 1187
35864124
LT
1188This command returns the length of the encoded text.
1189@end deffn
1190
1191@defun encode-coding-string string coding-system &optional nocopy
cc6d0d2c
RS
1192This function encodes the text in @var{string} according to coding
1193system @var{coding-system}. It returns a new string containing the
35864124
LT
1194encoded text, except when @var{nocopy} is non-@code{nil}, in which
1195case the function may return @var{string} itself if the encoding
1196operation is trivial. The result of encoding is a unibyte string.
cc6d0d2c
RS
1197@end defun
1198
35864124
LT
1199@deffn Command decode-coding-region start end coding-system
1200This command decodes the text from @var{start} to @var{end} according
cc6d0d2c
RS
1201to coding system @var{coding-system}. The decoded text replaces the
1202original text in the buffer. To make explicit decoding useful, the text
0ace421a
GM
1203before decoding ought to be a sequence of byte values, but both
1204multibyte and unibyte buffers are acceptable.
cc6d0d2c 1205
35864124
LT
1206This command returns the length of the decoded text.
1207@end deffn
1208
1209@defun decode-coding-string string coding-system &optional nocopy
cc6d0d2c
RS
1210This function decodes the text in @var{string} according to coding
1211system @var{coding-system}. It returns a new string containing the
35864124
LT
1212decoded text, except when @var{nocopy} is non-@code{nil}, in which
1213case the function may return @var{string} itself if the decoding
1214operation is trivial. To make explicit decoding useful, the contents
1215of @var{string} ought to be a sequence of byte values, but a multibyte
0ace421a 1216string is acceptable.
cc6d0d2c 1217@end defun
969fe9b5 1218
131bf943
RS
1219@defun decode-coding-inserted-region from to filename &optional visit beg end replace
1220This function decodes the text from @var{from} to @var{to} as if
1221it were being read from file @var{filename} using @code{insert-file-contents}
1222using the rest of the arguments provided.
1223
1224The normal way to use this function is after reading text from a file
1225without decoding, if you decide you would rather have decoded it.
1226Instead of deleting the text and reading it again, this time with
1227decoding, you can call this function.
1228@end defun
1229
a9f0a989
RS
1230@node Terminal I/O Encoding
1231@subsection Terminal I/O Encoding
1232
1233 Emacs can decode keyboard input using a coding system, and encode
2eb4136f
RS
1234terminal output. This is useful for terminals that transmit or display
1235text using a particular encoding such as Latin-1. Emacs does not set
1236@code{last-coding-system-used} for encoding or decoding for the
1237terminal.
a9f0a989
RS
1238
1239@defun keyboard-coding-system
a9f0a989
RS
1240This function returns the coding system that is in use for decoding
1241keyboard input---or @code{nil} if no coding system is to be used.
1242@end defun
1243
35864124
LT
1244@deffn Command set-keyboard-coding-system coding-system
1245This command specifies @var{coding-system} as the coding system to
a9f0a989
RS
1246use for decoding keyboard input. If @var{coding-system} is @code{nil},
1247that means do not decode keyboard input.
35864124 1248@end deffn
a9f0a989
RS
1249
1250@defun terminal-coding-system
a9f0a989
RS
1251This function returns the coding system that is in use for encoding
1252terminal output---or @code{nil} for no encoding.
1253@end defun
1254
35864124
LT
1255@deffn Command set-terminal-coding-system coding-system
1256This command specifies @var{coding-system} as the coding system to use
a9f0a989
RS
1257for encoding terminal output. If @var{coding-system} is @code{nil},
1258that means do not encode terminal output.
35864124 1259@end deffn
a9f0a989 1260
969fe9b5 1261@node MS-DOS File Types
a9f0a989 1262@subsection MS-DOS File Types
969fe9b5
RS
1263@cindex DOS file types
1264@cindex MS-DOS file types
1265@cindex Windows file types
1266@cindex file types on MS-DOS and Windows
1267@cindex text files and binary files
1268@cindex binary files and text files
1269
8241495d
RS
1270 On MS-DOS and Microsoft Windows, Emacs guesses the appropriate
1271end-of-line conversion for a file by looking at the file's name. This
0ace421a 1272feature classifies files as @dfn{text files} and @dfn{binary files}. By
8241495d
RS
1273``binary file'' we mean a file of literal byte values that are not
1274necessarily meant to be characters; Emacs does no end-of-line conversion
1275and no character code conversion for them. On the other hand, the bytes
1276in a text file are intended to represent characters; when you create a
1277new file whose name implies that it is a text file, Emacs uses DOS
1278end-of-line conversion.
969fe9b5
RS
1279
1280@defvar buffer-file-type
1281This variable, automatically buffer-local in each buffer, records the
a9f0a989
RS
1282file type of the buffer's visited file. When a buffer does not specify
1283a coding system with @code{buffer-file-coding-system}, this variable is
1284used to determine which coding system to use when writing the contents
1285of the buffer. It should be @code{nil} for text, @code{t} for binary.
1286If it is @code{t}, the coding system is @code{no-conversion}.
1287Otherwise, @code{undecided-dos} is used.
1288
1289Normally this variable is set by visiting a file; it is set to
1290@code{nil} if the file was visited without any actual conversion.
969fe9b5
RS
1291@end defvar
1292
1293@defopt file-name-buffer-file-type-alist
1294This variable holds an alist for recognizing text and binary files.
1295Each element has the form (@var{regexp} . @var{type}), where
1296@var{regexp} is matched against the file name, and @var{type} may be
1297@code{nil} for text, @code{t} for binary, or a function to call to
1298compute which. If it is a function, then it is called with a single
1299argument (the file name) and should return @code{t} or @code{nil}.
1300
8241495d 1301When running on MS-DOS or MS-Windows, Emacs checks this alist to decide
969fe9b5
RS
1302which coding system to use when reading a file. For a text file,
1303@code{undecided-dos} is used. For a binary file, @code{no-conversion}
1304is used.
1305
1306If no element in this alist matches a given file name, then
1307@code{default-buffer-file-type} says how to treat the file.
1308@end defopt
1309
1310@defopt default-buffer-file-type
1311This variable says how to handle files for which
1312@code{file-name-buffer-file-type-alist} says nothing about the type.
1313
1314If this variable is non-@code{nil}, then these files are treated as
a9f0a989
RS
1315binary: the coding system @code{no-conversion} is used. Otherwise,
1316nothing special is done for them---the coding system is deduced solely
1317from the file contents, in the usual Emacs fashion.
969fe9b5
RS
1318@end defopt
1319
a9f0a989
RS
1320@node Input Methods
1321@section Input Methods
1322@cindex input methods
1323
ad800164 1324 @dfn{Input methods} provide convenient ways of entering non-@acronym{ASCII}
a9f0a989 1325characters from the keyboard. Unlike coding systems, which translate
ad800164 1326non-@acronym{ASCII} characters to and from encodings meant to be read by
a9f0a989
RS
1327programs, input methods provide human-friendly commands. (@xref{Input
1328Methods,,, emacs, The GNU Emacs Manual}, for information on how users
1329use input methods to enter text.) How to define input methods is not
1330yet documented in this manual, but here we describe how to use them.
1331
1332 Each input method has a name, which is currently a string;
1333in the future, symbols may also be usable as input method names.
1334
a9f0a989
RS
1335@defvar current-input-method
1336This variable holds the name of the input method now active in the
1337current buffer. (It automatically becomes local in each buffer when set
1338in any fashion.) It is @code{nil} if no input method is active in the
1339buffer now.
969fe9b5
RS
1340@end defvar
1341
35864124 1342@defopt default-input-method
a9f0a989
RS
1343This variable holds the default input method for commands that choose an
1344input method. Unlike @code{current-input-method}, this variable is
1345normally global.
35864124 1346@end defopt
a9f0a989 1347
35864124
LT
1348@deffn Command set-input-method input-method
1349This command activates input method @var{input-method} for the current
a9f0a989 1350buffer. It also sets @code{default-input-method} to @var{input-method}.
35864124 1351If @var{input-method} is @code{nil}, this command deactivates any input
a9f0a989 1352method for the current buffer.
35864124 1353@end deffn
a9f0a989 1354
a9f0a989
RS
1355@defun read-input-method-name prompt &optional default inhibit-null
1356This function reads an input method name with the minibuffer, prompting
1357with @var{prompt}. If @var{default} is non-@code{nil}, that is returned
1358by default, if the user enters empty input. However, if
1359@var{inhibit-null} is non-@code{nil}, empty input signals an error.
1360
1361The returned value is a string.
1362@end defun
1363
a9f0a989
RS
1364@defvar input-method-alist
1365This variable defines all the supported input methods.
1366Each element defines one input method, and should have the form:
1367
1368@example
1911e6e5
RS
1369(@var{input-method} @var{language-env} @var{activate-func}
1370 @var{title} @var{description} @var{args}...)
a9f0a989
RS
1371@end example
1372
1911e6e5
RS
1373Here @var{input-method} is the input method name, a string;
1374@var{language-env} is another string, the name of the language
1375environment this input method is recommended for. (That serves only for
1376documentation purposes.)
a9f0a989 1377
a9f0a989
RS
1378@var{activate-func} is a function to call to activate this method. The
1379@var{args}, if any, are passed as arguments to @var{activate-func}. All
1380told, the arguments to @var{activate-func} are @var{input-method} and
1381the @var{args}.
0ace421a
GM
1382
1383@var{title} is a string to display in the mode line while this method is
1384active. @var{description} is a string describing this method and what
1385it is good for.
1911e6e5 1386@end defvar
a9f0a989 1387
2eb4136f 1388 The fundamental interface to input methods is through the
35864124
LT
1389variable @code{input-method-function}. @xref{Reading One Event},
1390and @ref{Invoking the Input Method}.
2468d0c0
DL
1391
1392@node Locales
1393@section Locales
1394@cindex locale
1395
1396 POSIX defines a concept of ``locales'' which control which language
1397to use in language-related features. These Emacs variables control
1398how Emacs interacts with these features.
1399
1400@defvar locale-coding-system
1401@tindex locale-coding-system
a007679c 1402@cindex keyboard input decoding on X
2468d0c0 1403This variable specifies the coding system to use for decoding system
a007679c
EZ
1404error messages and---on X Window system only---keyboard input, for
1405encoding the format argument to @code{format-time-string}, and for
1406decoding the return value of @code{format-time-string}.
2468d0c0
DL
1407@end defvar
1408
1409@defvar system-messages-locale
1410@tindex system-messages-locale
1411This variable specifies the locale to use for generating system error
1412messages. Changing the locale can cause messages to come out in a
9c17f494 1413different language or in a different orthography. If the variable is
2468d0c0
DL
1414@code{nil}, the locale is specified by environment variables in the
1415usual POSIX fashion.
1416@end defvar
1417
1418@defvar system-time-locale
1419@tindex system-time-locale
1420This variable specifies the locale to use for formatting time values.
1421Changing the locale can cause messages to appear according to the
1422conventions of a different language. If the variable is @code{nil}, the
1423locale is specified by environment variables in the usual POSIX fashion.
1424@end defvar
0ace421a 1425
131bf943
RS
1426@defun locale-info item
1427This function returns locale data @var{item} for the current POSIX
1428locale, if available. @var{item} should be one of these symbols:
1429
1430@table @code
1431@item codeset
1432Return the character set as a string (locale item @code{CODESET}).
1433
1434@item days
1435Return a 7-element vector of day names (locale items
1436@code{DAY_1} through @code{DAY_7});
1437
1438@item months
1439Return a 12-element vector of month names (locale items @code{MON_1}
1440through @code{MON_12}).
1441
1442@item paper
1443Return a list @code{(@var{width} @var{height})} for the default paper
35864124 1444size measured in millimeters (locale items @code{PAPER_WIDTH} and
131bf943
RS
1445@code{PAPER_HEIGHT}).
1446@end table
1447
1448If the system can't provide the requested information, or if
1449@var{item} is not one of those symbols, the value is @code{nil}. All
1450strings in the return value are decoded using
35864124 1451@code{locale-coding-system}. @xref{Locales,,, libc, The GNU Libc Manual},
131bf943
RS
1452for more information about locales and locale items.
1453@end defun
ab5796a9
MB
1454
1455@ignore
1456 arch-tag: be705bf8-941b-4c35-84fc-ad7d20ddb7cb
1457@end ignore