| 1 | @c -*-texinfo-*- |
| 2 | @c This is part of the GNU Emacs Lisp Reference Manual. |
| 3 | @c Copyright (C) 1998, 1999 Free Software Foundation, Inc. |
| 4 | @c See the file elisp.texi for copying conditions. |
| 5 | @setfilename ../info/characters |
| 6 | @node Non-ASCII Characters, Searching and Matching, Text, Top |
| 7 | @chapter Non-@sc{ascii} Characters |
| 8 | @cindex multibyte characters |
| 9 | @cindex non-@sc{ascii} characters |
| 10 | |
| 11 | This chapter covers the special issues relating to non-@sc{ascii} |
| 12 | characters and how they are stored in strings and buffers. |
| 13 | |
| 14 | @menu |
| 15 | * Text Representations:: Unibyte and multibyte representations |
| 16 | * Converting Representations:: Converting unibyte to multibyte and vice versa. |
| 17 | * Selecting a Representation:: Treating a byte sequence as unibyte or multi. |
| 18 | * Character Codes:: How unibyte and multibyte relate to |
| 19 | codes of individual characters. |
| 20 | * Character Sets:: The space of possible characters codes |
| 21 | is divided into various character sets. |
| 22 | * Chars and Bytes:: More information about multibyte encodings. |
| 23 | * Splitting Characters:: Converting a character to its byte sequence. |
| 24 | * Scanning Charsets:: Which character sets are used in a buffer? |
| 25 | * Translation of Characters:: Translation tables are used for conversion. |
| 26 | * Coding Systems:: Coding systems are conversions for saving files. |
| 27 | * Input Methods:: Input methods allow users to enter various |
| 28 | non-ASCII characters without special keyboards. |
| 29 | * Locales:: Interacting with the POSIX locale. |
| 30 | @end menu |
| 31 | |
| 32 | @node Text Representations |
| 33 | @section Text Representations |
| 34 | @cindex text representations |
| 35 | |
| 36 | Emacs has two @dfn{text representations}---two ways to represent text |
| 37 | in a string or buffer. These are called @dfn{unibyte} and |
| 38 | @dfn{multibyte}. Each string, and each buffer, uses one of these two |
| 39 | representations. For most purposes, you can ignore the issue of |
| 40 | representations, because Emacs converts text between them as |
| 41 | appropriate. Occasionally in Lisp programming you will need to pay |
| 42 | attention to the difference. |
| 43 | |
| 44 | @cindex unibyte text |
| 45 | In unibyte representation, each character occupies one byte and |
| 46 | therefore the possible character codes range from 0 to 255. Codes 0 |
| 47 | through 127 are @sc{ascii} characters; the codes from 128 through 255 |
| 48 | are used for one non-@sc{ascii} character set (you can choose which |
| 49 | character set by setting the variable @code{nonascii-insert-offset}). |
| 50 | |
| 51 | @cindex leading code |
| 52 | @cindex multibyte text |
| 53 | @cindex trailing codes |
| 54 | In multibyte representation, a character may occupy more than one |
| 55 | byte, and as a result, the full range of Emacs character codes can be |
| 56 | stored. The first byte of a multibyte character is always in the range |
| 57 | 128 through 159 (octal 0200 through 0237). These values are called |
| 58 | @dfn{leading codes}. The second and subsequent bytes of a multibyte |
| 59 | character are always in the range 160 through 255 (octal 0240 through |
| 60 | 0377); these values are @dfn{trailing codes}. |
| 61 | |
| 62 | Some sequences of bytes are not valid in multibyte text: for example, |
| 63 | a single isolated byte in the range 128 through 159 is not allowed. But |
| 64 | character codes 128 through 159 can appear in multibyte text, |
| 65 | represented as two-byte sequences. All the character codes 128 through |
| 66 | 255 are possible (though slightly abnormal) in multibyte text; they |
| 67 | appear in multibyte buffers and strings when you do explicit encoding |
| 68 | and decoding (@pxref{Explicit Encoding}). |
| 69 | |
| 70 | In a buffer, the buffer-local value of the variable |
| 71 | @code{enable-multibyte-characters} specifies the representation used. |
| 72 | The representation for a string is determined and recorded in the string |
| 73 | when the string is constructed. |
| 74 | |
| 75 | @defvar enable-multibyte-characters |
| 76 | This variable specifies the current buffer's text representation. |
| 77 | If it is non-@code{nil}, the buffer contains multibyte text; otherwise, |
| 78 | it contains unibyte text. |
| 79 | |
| 80 | You cannot set this variable directly; instead, use the function |
| 81 | @code{set-buffer-multibyte} to change a buffer's representation. |
| 82 | @end defvar |
| 83 | |
| 84 | @defvar default-enable-multibyte-characters |
| 85 | This variable's value is entirely equivalent to @code{(default-value |
| 86 | 'enable-multibyte-characters)}, and setting this variable changes that |
| 87 | default value. Setting the local binding of |
| 88 | @code{enable-multibyte-characters} in a specific buffer is not allowed, |
| 89 | but changing the default value is supported, and it is a reasonable |
| 90 | thing to do, because it has no effect on existing buffers. |
| 91 | |
| 92 | The @samp{--unibyte} command line option does its job by setting the |
| 93 | default value to @code{nil} early in startup. |
| 94 | @end defvar |
| 95 | |
| 96 | @defun position-bytes position |
| 97 | @tindex position-bytes |
| 98 | Return the byte-position corresponding to buffer position @var{position} |
| 99 | in the current buffer. |
| 100 | @end defun |
| 101 | |
| 102 | @defun byte-to-position byte-position |
| 103 | @tindex byte-to-position |
| 104 | Return the buffer position corresponding to byte-position |
| 105 | @var{byte-position} in the current buffer. |
| 106 | @end defun |
| 107 | |
| 108 | @defun multibyte-string-p string |
| 109 | Return @code{t} if @var{string} is a multibyte string. |
| 110 | @end defun |
| 111 | |
| 112 | @node Converting Representations |
| 113 | @section Converting Text Representations |
| 114 | |
| 115 | Emacs can convert unibyte text to multibyte; it can also convert |
| 116 | multibyte text to unibyte, though this conversion loses information. In |
| 117 | general these conversions happen when inserting text into a buffer, or |
| 118 | when putting text from several strings together in one string. You can |
| 119 | also explicitly convert a string's contents to either representation. |
| 120 | |
| 121 | Emacs chooses the representation for a string based on the text that |
| 122 | it is constructed from. The general rule is to convert unibyte text to |
| 123 | multibyte text when combining it with other multibyte text, because the |
| 124 | multibyte representation is more general and can hold whatever |
| 125 | characters the unibyte text has. |
| 126 | |
| 127 | When inserting text into a buffer, Emacs converts the text to the |
| 128 | buffer's representation, as specified by |
| 129 | @code{enable-multibyte-characters} in that buffer. In particular, when |
| 130 | you insert multibyte text into a unibyte buffer, Emacs converts the text |
| 131 | to unibyte, even though this conversion cannot in general preserve all |
| 132 | the characters that might be in the multibyte text. The other natural |
| 133 | alternative, to convert the buffer contents to multibyte, is not |
| 134 | acceptable because the buffer's representation is a choice made by the |
| 135 | user that cannot be overridden automatically. |
| 136 | |
| 137 | Converting unibyte text to multibyte text leaves @sc{ascii} characters |
| 138 | unchanged, and likewise character codes 128 through 159. It converts |
| 139 | the non-@sc{ascii} codes 160 through 255 by adding the value |
| 140 | @code{nonascii-insert-offset} to each character code. By setting this |
| 141 | variable, you specify which character set the unibyte characters |
| 142 | correspond to (@pxref{Character Sets}). For example, if |
| 143 | @code{nonascii-insert-offset} is 2048, which is @code{(- (make-char |
| 144 | 'latin-iso8859-1) 128)}, then the unibyte non-@sc{ascii} characters |
| 145 | correspond to Latin 1. If it is 2688, which is @code{(- (make-char |
| 146 | 'greek-iso8859-7) 128)}, then they correspond to Greek letters. |
| 147 | |
| 148 | Converting multibyte text to unibyte is simpler: it discards all but |
| 149 | the low 8 bits of each character code. If @code{nonascii-insert-offset} |
| 150 | has a reasonable value, corresponding to the beginning of some character |
| 151 | set, this conversion is the inverse of the other: converting unibyte |
| 152 | text to multibyte and back to unibyte reproduces the original unibyte |
| 153 | text. |
| 154 | |
| 155 | @defvar nonascii-insert-offset |
| 156 | This variable specifies the amount to add to a non-@sc{ascii} character |
| 157 | when converting unibyte text to multibyte. It also applies when |
| 158 | @code{self-insert-command} inserts a character in the unibyte |
| 159 | non-@sc{ascii} range, 128 through 255. However, the functions |
| 160 | @code{insert} and @code{insert-char} do not perform this conversion. |
| 161 | |
| 162 | The right value to use to select character set @var{cs} is @code{(- |
| 163 | (make-char @var{cs}) 128)}. If the value of |
| 164 | @code{nonascii-insert-offset} is zero, then conversion actually uses the |
| 165 | value for the Latin 1 character set, rather than zero. |
| 166 | @end defvar |
| 167 | |
| 168 | @defvar nonascii-translation-table |
| 169 | This variable provides a more general alternative to |
| 170 | @code{nonascii-insert-offset}. You can use it to specify independently |
| 171 | how to translate each code in the range of 128 through 255 into a |
| 172 | multibyte character. The value should be a char-table, or @code{nil}. |
| 173 | If this is non-@code{nil}, it overrides @code{nonascii-insert-offset}. |
| 174 | @end defvar |
| 175 | |
| 176 | @defun string-make-unibyte string |
| 177 | This function converts the text of @var{string} to unibyte |
| 178 | representation, if it isn't already, and returns the result. If |
| 179 | @var{string} is a unibyte string, it is returned unchanged. |
| 180 | Multibyte character codes are converted to unibyte |
| 181 | by using just the low 8 bits. |
| 182 | @end defun |
| 183 | |
| 184 | @defun string-make-multibyte string |
| 185 | This function converts the text of @var{string} to multibyte |
| 186 | representation, if it isn't already, and returns the result. If |
| 187 | @var{string} is a multibyte string, it is returned unchanged. |
| 188 | The function @code{unibyte-char-to-multibyte} is used to convert |
| 189 | each unibyte character to a multibyte character. |
| 190 | @end defun |
| 191 | |
| 192 | @node Selecting a Representation |
| 193 | @section Selecting a Representation |
| 194 | |
| 195 | Sometimes it is useful to examine an existing buffer or string as |
| 196 | multibyte when it was unibyte, or vice versa. |
| 197 | |
| 198 | @defun set-buffer-multibyte multibyte |
| 199 | Set the representation type of the current buffer. If @var{multibyte} |
| 200 | is non-@code{nil}, the buffer becomes multibyte. If @var{multibyte} |
| 201 | is @code{nil}, the buffer becomes unibyte. |
| 202 | |
| 203 | This function leaves the buffer contents unchanged when viewed as a |
| 204 | sequence of bytes. As a consequence, it can change the contents viewed |
| 205 | as characters; a sequence of two bytes which is treated as one character |
| 206 | in multibyte representation will count as two characters in unibyte |
| 207 | representation. Character codes 128 through 159 are an exception. They |
| 208 | are represented by one byte in a unibyte buffer, but when the buffer is |
| 209 | set to multibyte, they are converted to two-byte sequences, and vice |
| 210 | versa. |
| 211 | |
| 212 | This function sets @code{enable-multibyte-characters} to record which |
| 213 | representation is in use. It also adjusts various data in the buffer |
| 214 | (including overlays, text properties and markers) so that they cover the |
| 215 | same text as they did before. |
| 216 | |
| 217 | You cannot use @code{set-buffer-multibyte} on an indirect buffer, |
| 218 | because indirect buffers always inherit the representation of the |
| 219 | base buffer. |
| 220 | @end defun |
| 221 | |
| 222 | @defun string-as-unibyte string |
| 223 | This function returns a string with the same bytes as @var{string} but |
| 224 | treating each byte as a character. This means that the value may have |
| 225 | more characters than @var{string} has. |
| 226 | |
| 227 | If @var{string} is already a unibyte string, then the value is |
| 228 | @var{string} itself. Otherwise it is a newly created string, with no |
| 229 | text properties. If @var{string} is multibyte, any characters it |
| 230 | contains of charset @var{eight-bit-control} or @var{eight-bit-graphic} |
| 231 | are converted to the corresponding single byte. |
| 232 | @end defun |
| 233 | |
| 234 | @defun string-as-multibyte string |
| 235 | This function returns a string with the same bytes as @var{string} but |
| 236 | treating each multibyte sequence as one character. This means that the |
| 237 | value may have fewer characters than @var{string} has. |
| 238 | |
| 239 | If @var{string} is already a multibyte string, then the value is |
| 240 | @var{string} itself. Otherwise it is a newly created string, with no |
| 241 | text properties. If @var{string} is unibyte and contains any individual |
| 242 | 8-bit bytes (i.e.@: not part of a multibyte form), they are converted to |
| 243 | the corresponding multibyte character of charset @var{eight-bit-control} |
| 244 | or @var{eight-bit-graphic}. |
| 245 | @end defun |
| 246 | |
| 247 | @node Character Codes |
| 248 | @section Character Codes |
| 249 | @cindex character codes |
| 250 | |
| 251 | The unibyte and multibyte text representations use different character |
| 252 | codes. The valid character codes for unibyte representation range from |
| 253 | 0 to 255---the values that can fit in one byte. The valid character |
| 254 | codes for multibyte representation range from 0 to 524287, but not all |
| 255 | values in that range are valid. The values 128 through 255 are not |
| 256 | entirely proper in multibyte text, but they can occur if you do explicit |
| 257 | encoding and decoding (@pxref{Explicit Encoding}). Some other character |
| 258 | codes cannot occur at all in multibyte text. Only the @sc{ascii} codes |
| 259 | 0 through 127 are completely legitimate in both representations. |
| 260 | |
| 261 | @defun char-valid-p charcode &optional genericp |
| 262 | This returns @code{t} if @var{charcode} is valid for either one of the two |
| 263 | text representations. |
| 264 | |
| 265 | @example |
| 266 | (char-valid-p 65) |
| 267 | @result{} t |
| 268 | (char-valid-p 256) |
| 269 | @result{} nil |
| 270 | (char-valid-p 2248) |
| 271 | @result{} t |
| 272 | @end example |
| 273 | |
| 274 | If the optional argument @var{genericp} is non-nil, this function |
| 275 | returns @code{t} if @var{charcode} is a generic character |
| 276 | (@pxref{Splitting Characters}). |
| 277 | @end defun |
| 278 | |
| 279 | @node Character Sets |
| 280 | @section Character Sets |
| 281 | @cindex character sets |
| 282 | |
| 283 | Emacs classifies characters into various @dfn{character sets}, each of |
| 284 | which has a name which is a symbol. Each character belongs to one and |
| 285 | only one character set. |
| 286 | |
| 287 | In general, there is one character set for each distinct script. For |
| 288 | example, @code{latin-iso8859-1} is one character set, |
| 289 | @code{greek-iso8859-7} is another, and @code{ascii} is another. An |
| 290 | Emacs character set can hold at most 9025 characters; therefore, in some |
| 291 | cases, characters that would logically be grouped together are split |
| 292 | into several character sets. For example, one set of Chinese |
| 293 | characters, generally known as Big 5, is divided into two Emacs |
| 294 | character sets, @code{chinese-big5-1} and @code{chinese-big5-2}. |
| 295 | |
| 296 | @sc{ascii} characters are in character set @code{ascii}. The |
| 297 | non-@sc{ascii} characters 128 through 159 are in character set |
| 298 | @code{eight-bit-control}, and codes 160 through 255 are in character set |
| 299 | @code{eight-bit-graphic}. |
| 300 | |
| 301 | @defun charsetp object |
| 302 | Returns @code{t} if @var{object} is a symbol that names a character set, |
| 303 | @code{nil} otherwise. |
| 304 | @end defun |
| 305 | |
| 306 | @defun charset-list |
| 307 | This function returns a list of all defined character set names. |
| 308 | @end defun |
| 309 | |
| 310 | @defun char-charset character |
| 311 | This function returns the name of the character set that @var{character} |
| 312 | belongs to. |
| 313 | @end defun |
| 314 | |
| 315 | @defun charset-plist charset |
| 316 | @tindex charset-plist |
| 317 | This function returns the charset property list of the character set |
| 318 | @var{charset}. Although @var{charset} is a symbol, this is not the same |
| 319 | as the property list of that symbol. Charset properties are used for |
| 320 | special purposes within Emacs; for example, |
| 321 | @code{preferred-coding-system} helps determine which coding system to |
| 322 | use to encode characters in a charset. |
| 323 | @end defun |
| 324 | |
| 325 | @node Chars and Bytes |
| 326 | @section Characters and Bytes |
| 327 | @cindex bytes and characters |
| 328 | |
| 329 | @cindex introduction sequence |
| 330 | @cindex dimension (of character set) |
| 331 | In multibyte representation, each character occupies one or more |
| 332 | bytes. Each character set has an @dfn{introduction sequence}, which is |
| 333 | normally one or two bytes long. (Exception: the @sc{ascii} character |
| 334 | set and the @sc{eight-bit-graphic} character set have a zero-length |
| 335 | introduction sequence.) The introduction sequence is the beginning of |
| 336 | the byte sequence for any character in the character set. The rest of |
| 337 | the character's bytes distinguish it from the other characters in the |
| 338 | same character set. Depending on the character set, there are either |
| 339 | one or two distinguishing bytes; the number of such bytes is called the |
| 340 | @dfn{dimension} of the character set. |
| 341 | |
| 342 | @defun charset-dimension charset |
| 343 | This function returns the dimension of @var{charset}; at present, the |
| 344 | dimension is always 1 or 2. |
| 345 | @end defun |
| 346 | |
| 347 | @defun charset-bytes charset |
| 348 | @tindex charset-bytes |
| 349 | This function returns the number of bytes used to represent a character |
| 350 | in character set @var{charset}. |
| 351 | @end defun |
| 352 | |
| 353 | This is the simplest way to determine the byte length of a character |
| 354 | set's introduction sequence: |
| 355 | |
| 356 | @example |
| 357 | (- (charset-bytes @var{charset}) |
| 358 | (charset-dimension @var{charset})) |
| 359 | @end example |
| 360 | |
| 361 | @node Splitting Characters |
| 362 | @section Splitting Characters |
| 363 | |
| 364 | The functions in this section convert between characters and the byte |
| 365 | values used to represent them. For most purposes, there is no need to |
| 366 | be concerned with the sequence of bytes used to represent a character, |
| 367 | because Emacs translates automatically when necessary. |
| 368 | |
| 369 | @defun split-char character |
| 370 | Return a list containing the name of the character set of |
| 371 | @var{character}, followed by one or two byte values (integers) which |
| 372 | identify @var{character} within that character set. The number of byte |
| 373 | values is the character set's dimension. |
| 374 | |
| 375 | @example |
| 376 | (split-char 2248) |
| 377 | @result{} (latin-iso8859-1 72) |
| 378 | (split-char 65) |
| 379 | @result{} (ascii 65) |
| 380 | (split-char 128) |
| 381 | @result{} (eight-bit-control 128) |
| 382 | @end example |
| 383 | @end defun |
| 384 | |
| 385 | @defun make-char charset &optional code1 code2 |
| 386 | This function returns the character in character set @var{charset} whose |
| 387 | position codes are @var{code1} and @var{code2}. This is roughly the |
| 388 | inverse of @code{split-char}. Normally, you should specify either one |
| 389 | or both of @var{code1} and @var{code2} according to the dimension of |
| 390 | @var{charset}. For example, |
| 391 | |
| 392 | @example |
| 393 | (make-char 'latin-iso8859-1 72) |
| 394 | @result{} 2248 |
| 395 | @end example |
| 396 | @end defun |
| 397 | |
| 398 | @cindex generic characters |
| 399 | If you call @code{make-char} with no @var{byte-values}, the result is |
| 400 | a @dfn{generic character} which stands for @var{charset}. A generic |
| 401 | character is an integer, but it is @emph{not} valid for insertion in the |
| 402 | buffer as a character. It can be used in @code{char-table-range} to |
| 403 | refer to the whole character set (@pxref{Char-Tables}). |
| 404 | @code{char-valid-p} returns @code{nil} for generic characters. |
| 405 | For example: |
| 406 | |
| 407 | @example |
| 408 | (make-char 'latin-iso8859-1) |
| 409 | @result{} 2176 |
| 410 | (char-valid-p 2176) |
| 411 | @result{} nil |
| 412 | (char-valid-p 2176 t) |
| 413 | @result{} t |
| 414 | (split-char 2176) |
| 415 | @result{} (latin-iso8859-1 0) |
| 416 | @end example |
| 417 | |
| 418 | The character sets @sc{ascii}, @sc{eight-bit-control}, and |
| 419 | @sc{eight-bit-graphic} don't have corresponding generic characters. If |
| 420 | @var{charset} is one of them and you don't supply @var{code1}, |
| 421 | @code{make-char} returns the character code corresponding to the |
| 422 | smallest code in @var{charset}. |
| 423 | |
| 424 | @node Scanning Charsets |
| 425 | @section Scanning for Character Sets |
| 426 | |
| 427 | Sometimes it is useful to find out which character sets appear in a |
| 428 | part of a buffer or a string. One use for this is in determining which |
| 429 | coding systems (@pxref{Coding Systems}) are capable of representing all |
| 430 | of the text in question. |
| 431 | |
| 432 | @defun find-charset-region beg end &optional translation |
| 433 | This function returns a list of the character sets that appear in the |
| 434 | current buffer between positions @var{beg} and @var{end}. |
| 435 | |
| 436 | The optional argument @var{translation} specifies a translation table to |
| 437 | be used in scanning the text (@pxref{Translation of Characters}). If it |
| 438 | is non-@code{nil}, then each character in the region is translated |
| 439 | through this table, and the value returned describes the translated |
| 440 | characters instead of the characters actually in the buffer. |
| 441 | @end defun |
| 442 | |
| 443 | @defun find-charset-string string &optional translation |
| 444 | This function returns a list of the character sets that appear in the |
| 445 | string @var{string}. It is just like @code{find-charset-region}, except |
| 446 | that it applies to the contents of @var{string} instead of part of the |
| 447 | current buffer. |
| 448 | @end defun |
| 449 | |
| 450 | @node Translation of Characters |
| 451 | @section Translation of Characters |
| 452 | @cindex character translation tables |
| 453 | @cindex translation tables |
| 454 | |
| 455 | A @dfn{translation table} specifies a mapping of characters |
| 456 | into characters. These tables are used in encoding and decoding, and |
| 457 | for other purposes. Some coding systems specify their own particular |
| 458 | translation tables; there are also default translation tables which |
| 459 | apply to all other coding systems. |
| 460 | |
| 461 | @defun make-translation-table &rest translations |
| 462 | This function returns a translation table based on the argument |
| 463 | @var{translations}. Each element of @var{translations} should be a |
| 464 | list of elements of the form @code{(@var{from} . @var{to})}; this says |
| 465 | to translate the character @var{from} into @var{to}. |
| 466 | |
| 467 | The arguments and the forms in each argument are processed in order, |
| 468 | and if a previous form already translates @var{to} to some other |
| 469 | character, say @var{to-alt}, @var{from} is also translated to |
| 470 | @var{to-alt}. |
| 471 | |
| 472 | You can also map one whole character set into another character set with |
| 473 | the same dimension. To do this, you specify a generic character (which |
| 474 | designates a character set) for @var{from} (@pxref{Splitting Characters}). |
| 475 | In this case, @var{to} should also be a generic character, for another |
| 476 | character set of the same dimension. Then the translation table |
| 477 | translates each character of @var{from}'s character set into the |
| 478 | corresponding character of @var{to}'s character set. |
| 479 | @end defun |
| 480 | |
| 481 | In decoding, the translation table's translations are applied to the |
| 482 | characters that result from ordinary decoding. If a coding system has |
| 483 | property @code{character-translation-table-for-decode}, that specifies |
| 484 | the translation table to use. Otherwise, if |
| 485 | @code{standard-translation-table-for-decode} is non-@code{nil}, decoding |
| 486 | uses that table. |
| 487 | |
| 488 | In encoding, the translation table's translations are applied to the |
| 489 | characters in the buffer, and the result of translation is actually |
| 490 | encoded. If a coding system has property |
| 491 | @code{character-translation-table-for-encode}, that specifies the |
| 492 | translation table to use. Otherwise the variable |
| 493 | @code{standard-translation-table-for-encode} specifies the translation |
| 494 | table. |
| 495 | |
| 496 | @defvar standard-translation-table-for-decode |
| 497 | This is the default translation table for decoding, for |
| 498 | coding systems that don't specify any other translation table. |
| 499 | @end defvar |
| 500 | |
| 501 | @defvar standard-translation-table-for-encode |
| 502 | This is the default translation table for encoding, for |
| 503 | coding systems that don't specify any other translation table. |
| 504 | @end defvar |
| 505 | |
| 506 | @node Coding Systems |
| 507 | @section Coding Systems |
| 508 | |
| 509 | @cindex coding system |
| 510 | When Emacs reads or writes a file, and when Emacs sends text to a |
| 511 | subprocess or receives text from a subprocess, it normally performs |
| 512 | character code conversion and end-of-line conversion as specified |
| 513 | by a particular @dfn{coding system}. |
| 514 | |
| 515 | How to define a coding system is an arcane matter, and is not |
| 516 | documented here. |
| 517 | |
| 518 | @menu |
| 519 | * Coding System Basics:: Basic concepts. |
| 520 | * Encoding and I/O:: How file I/O functions handle coding systems. |
| 521 | * Lisp and Coding Systems:: Functions to operate on coding system names. |
| 522 | * User-Chosen Coding Systems:: Asking the user to choose a coding system. |
| 523 | * Default Coding Systems:: Controlling the default choices. |
| 524 | * Specifying Coding Systems:: Requesting a particular coding system |
| 525 | for a single file operation. |
| 526 | * Explicit Encoding:: Encoding or decoding text without doing I/O. |
| 527 | * Terminal I/O Encoding:: Use of encoding for terminal I/O. |
| 528 | * MS-DOS File Types:: How DOS "text" and "binary" files |
| 529 | relate to coding systems. |
| 530 | @end menu |
| 531 | |
| 532 | @node Coding System Basics |
| 533 | @subsection Basic Concepts of Coding Systems |
| 534 | |
| 535 | @cindex character code conversion |
| 536 | @dfn{Character code conversion} involves conversion between the encoding |
| 537 | used inside Emacs and some other encoding. Emacs supports many |
| 538 | different encodings, in that it can convert to and from them. For |
| 539 | example, it can convert text to or from encodings such as Latin 1, Latin |
| 540 | 2, Latin 3, Latin 4, Latin 5, and several variants of ISO 2022. In some |
| 541 | cases, Emacs supports several alternative encodings for the same |
| 542 | characters; for example, there are three coding systems for the Cyrillic |
| 543 | (Russian) alphabet: ISO, Alternativnyj, and KOI8. |
| 544 | |
| 545 | Most coding systems specify a particular character code for |
| 546 | conversion, but some of them leave the choice unspecified---to be chosen |
| 547 | heuristically for each file, based on the data. |
| 548 | |
| 549 | @cindex end of line conversion |
| 550 | @dfn{End of line conversion} handles three different conventions used |
| 551 | on various systems for representing end of line in files. The Unix |
| 552 | convention is to use the linefeed character (also called newline). The |
| 553 | DOS convention is to use a carriage-return and a linefeed at the end of |
| 554 | a line. The Mac convention is to use just carriage-return. |
| 555 | |
| 556 | @cindex base coding system |
| 557 | @cindex variant coding system |
| 558 | @dfn{Base coding systems} such as @code{latin-1} leave the end-of-line |
| 559 | conversion unspecified, to be chosen based on the data. @dfn{Variant |
| 560 | coding systems} such as @code{latin-1-unix}, @code{latin-1-dos} and |
| 561 | @code{latin-1-mac} specify the end-of-line conversion explicitly as |
| 562 | well. Most base coding systems have three corresponding variants whose |
| 563 | names are formed by adding @samp{-unix}, @samp{-dos} and @samp{-mac}. |
| 564 | |
| 565 | The coding system @code{raw-text} is special in that it prevents |
| 566 | character code conversion, and causes the buffer visited with that |
| 567 | coding system to be a unibyte buffer. It does not specify the |
| 568 | end-of-line conversion, allowing that to be determined as usual by the |
| 569 | data, and has the usual three variants which specify the end-of-line |
| 570 | conversion. @code{no-conversion} is equivalent to @code{raw-text-unix}: |
| 571 | it specifies no conversion of either character codes or end-of-line. |
| 572 | |
| 573 | The coding system @code{emacs-mule} specifies that the data is |
| 574 | represented in the internal Emacs encoding. This is like |
| 575 | @code{raw-text} in that no code conversion happens, but different in |
| 576 | that the result is multibyte data. |
| 577 | |
| 578 | @defun coding-system-get coding-system property |
| 579 | This function returns the specified property of the coding system |
| 580 | @var{coding-system}. Most coding system properties exist for internal |
| 581 | purposes, but one that you might find useful is @code{mime-charset}. |
| 582 | That property's value is the name used in MIME for the character coding |
| 583 | which this coding system can read and write. Examples: |
| 584 | |
| 585 | @example |
| 586 | (coding-system-get 'iso-latin-1 'mime-charset) |
| 587 | @result{} iso-8859-1 |
| 588 | (coding-system-get 'iso-2022-cn 'mime-charset) |
| 589 | @result{} iso-2022-cn |
| 590 | (coding-system-get 'cyrillic-koi8 'mime-charset) |
| 591 | @result{} koi8-r |
| 592 | @end example |
| 593 | |
| 594 | The value of the @code{mime-charset} property is also defined |
| 595 | as an alias for the coding system. |
| 596 | @end defun |
| 597 | |
| 598 | @node Encoding and I/O |
| 599 | @subsection Encoding and I/O |
| 600 | |
| 601 | The principal purpose of coding systems is for use in reading and |
| 602 | writing files. The function @code{insert-file-contents} uses |
| 603 | a coding system for decoding the file data, and @code{write-region} |
| 604 | uses one to encode the buffer contents. |
| 605 | |
| 606 | You can specify the coding system to use either explicitly |
| 607 | (@pxref{Specifying Coding Systems}), or implicitly using the defaulting |
| 608 | mechanism (@pxref{Default Coding Systems}). But these methods may not |
| 609 | completely specify what to do. For example, they may choose a coding |
| 610 | system such as @code{undefined} which leaves the character code |
| 611 | conversion to be determined from the data. In these cases, the I/O |
| 612 | operation finishes the job of choosing a coding system. Very often |
| 613 | you will want to find out afterwards which coding system was chosen. |
| 614 | |
| 615 | @defvar buffer-file-coding-system |
| 616 | This variable records the coding system that was used for visiting the |
| 617 | current buffer. It is used for saving the buffer, and for writing part |
| 618 | of the buffer with @code{write-region}. When those operations ask the |
| 619 | user to specify a different coding system, |
| 620 | @code{buffer-file-coding-system} is updated to the coding system |
| 621 | specified. |
| 622 | |
| 623 | However, @code{buffer-file-coding-system} does not affect sending text |
| 624 | to a subprocess. |
| 625 | @end defvar |
| 626 | |
| 627 | @defvar save-buffer-coding-system |
| 628 | This variable specifies the coding system for saving the buffer (by |
| 629 | overriding @code{buffer-file-coding-system}). Note that it is not used |
| 630 | for @code{write-region}. |
| 631 | |
| 632 | When a command to save the buffer starts out to use |
| 633 | @code{buffer-file-coding-system} (or @code{save-buffer-coding-system}), |
| 634 | and that coding system cannot handle |
| 635 | the actual text in the buffer, the command asks the user to choose |
| 636 | another coding system. After that happens, the command also updates |
| 637 | @code{buffer-file-coding-system} to represent the coding system that the |
| 638 | user specified. |
| 639 | @end defvar |
| 640 | |
| 641 | @defvar last-coding-system-used |
| 642 | I/O operations for files and subprocesses set this variable to the |
| 643 | coding system name that was used. The explicit encoding and decoding |
| 644 | functions (@pxref{Explicit Encoding}) set it too. |
| 645 | |
| 646 | @strong{Warning:} Since receiving subprocess output sets this variable, |
| 647 | it can change whenever Emacs waits; therefore, you should copy the |
| 648 | value shortly after the function call that stores the value you are |
| 649 | interested in. |
| 650 | @end defvar |
| 651 | |
| 652 | The variable @code{selection-coding-system} specifies how to encode |
| 653 | selections for the window system. @xref{Window System Selections}. |
| 654 | |
| 655 | @node Lisp and Coding Systems |
| 656 | @subsection Coding Systems in Lisp |
| 657 | |
| 658 | Here are the Lisp facilities for working with coding systems: |
| 659 | |
| 660 | @defun coding-system-list &optional base-only |
| 661 | This function returns a list of all coding system names (symbols). If |
| 662 | @var{base-only} is non-@code{nil}, the value includes only the |
| 663 | base coding systems. Otherwise, it includes alias and variant coding |
| 664 | systems as well. |
| 665 | @end defun |
| 666 | |
| 667 | @defun coding-system-p object |
| 668 | This function returns @code{t} if @var{object} is a coding system |
| 669 | name. |
| 670 | @end defun |
| 671 | |
| 672 | @defun check-coding-system coding-system |
| 673 | This function checks the validity of @var{coding-system}. |
| 674 | If that is valid, it returns @var{coding-system}. |
| 675 | Otherwise it signals an error with condition @code{coding-system-error}. |
| 676 | @end defun |
| 677 | |
| 678 | @defun coding-system-change-eol-conversion coding-system eol-type |
| 679 | This function returns a coding system which is like @var{coding-system} |
| 680 | except for its eol conversion, which is specified by @code{eol-type}. |
| 681 | @var{eol-type} should be @code{unix}, @code{dos}, @code{mac}, or |
| 682 | @code{nil}. If it is @code{nil}, the returned coding system determines |
| 683 | the end-of-line conversion from the data. |
| 684 | @end defun |
| 685 | |
| 686 | @defun coding-system-change-text-conversion eol-coding text-coding |
| 687 | This function returns a coding system which uses the end-of-line |
| 688 | conversion of @var{eol-coding}, and the text conversion of |
| 689 | @var{text-coding}. If @var{text-coding} is @code{nil}, it returns |
| 690 | @code{undecided}, or one of its variants according to @var{eol-coding}. |
| 691 | @end defun |
| 692 | |
| 693 | @defun find-coding-systems-region from to |
| 694 | This function returns a list of coding systems that could be used to |
| 695 | encode a text between @var{from} and @var{to}. All coding systems in |
| 696 | the list can safely encode any multibyte characters in that portion of |
| 697 | the text. |
| 698 | |
| 699 | If the text contains no multibyte characters, the function returns the |
| 700 | list @code{(undecided)}. |
| 701 | @end defun |
| 702 | |
| 703 | @defun find-coding-systems-string string |
| 704 | This function returns a list of coding systems that could be used to |
| 705 | encode the text of @var{string}. All coding systems in the list can |
| 706 | safely encode any multibyte characters in @var{string}. If the text |
| 707 | contains no multibyte characters, this returns the list |
| 708 | @code{(undecided)}. |
| 709 | @end defun |
| 710 | |
| 711 | @defun find-coding-systems-for-charsets charsets |
| 712 | This function returns a list of coding systems that could be used to |
| 713 | encode all the character sets in the list @var{charsets}. |
| 714 | @end defun |
| 715 | |
| 716 | @defun detect-coding-region start end &optional highest |
| 717 | This function chooses a plausible coding system for decoding the text |
| 718 | from @var{start} to @var{end}. This text should be a byte sequence |
| 719 | (@pxref{Explicit Encoding}). |
| 720 | |
| 721 | Normally this function returns a list of coding systems that could |
| 722 | handle decoding the text that was scanned. They are listed in order of |
| 723 | decreasing priority. But if @var{highest} is non-@code{nil}, then the |
| 724 | return value is just one coding system, the one that is highest in |
| 725 | priority. |
| 726 | |
| 727 | If the region contains only @sc{ascii} characters, the value |
| 728 | is @code{undecided} or @code{(undecided)}. |
| 729 | @end defun |
| 730 | |
| 731 | @defun detect-coding-string string highest |
| 732 | This function is like @code{detect-coding-region} except that it |
| 733 | operates on the contents of @var{string} instead of bytes in the buffer. |
| 734 | @end defun |
| 735 | |
| 736 | @xref{Process Information}, for how to examine or set the coding |
| 737 | systems used for I/O to a subprocess. |
| 738 | |
| 739 | @node User-Chosen Coding Systems |
| 740 | @subsection User-Chosen Coding Systems |
| 741 | |
| 742 | @defun select-safe-coding-system from to &optional default-coding-system accept-default-p |
| 743 | This function selects a coding system for encoding specified text, |
| 744 | asking the user to choose if necessary. Normally the specified text |
| 745 | is the text in the current buffer between @var{from} and @var{to}, |
| 746 | defaulting to the whole buffer if they are @code{nil}. If @var{from} |
| 747 | is a string, the string specifies the text to encode, and @var{to} is |
| 748 | ignored. |
| 749 | |
| 750 | If @var{default-coding-system} is non-@code{nil}, that is the first |
| 751 | coding system to try; if that can handle the text, |
| 752 | @code{select-safe-coding-system} returns that coding system. It can |
| 753 | also be a list of coding systems; then the function tries each of them |
| 754 | one by one. After trying all of them, it next tries the user's most |
| 755 | preferred coding system (@pxref{Recognize Coding, |
| 756 | prefer-coding-system, the description of @code{prefer-coding-system}, |
| 757 | emacs, GNU Emacs Manual}), and after that the current buffer's value |
| 758 | of @code{buffer-file-coding-system} (if it is not @code{undecided}). |
| 759 | |
| 760 | If one of those coding systems can safely encode all the specified |
| 761 | text, @code{select-safe-coding-system} chooses it and returns it. |
| 762 | Otherwise, it asks the user to choose from a list of coding systems |
| 763 | which can encode all the text, and returns the user's choice. |
| 764 | |
| 765 | The optional argument @var{accept-default-p}, if non-@code{nil}, |
| 766 | should be a function to determine whether the coding system selected |
| 767 | without user interaction is acceptable. If this function returns |
| 768 | @code{nil}, the silently selected coding system is rejected, and the |
| 769 | user is asked to select a coding system from a list of possible |
| 770 | candidates. |
| 771 | |
| 772 | @vindex select-safe-coding-system-accept-default-p |
| 773 | If the variable @code{select-safe-coding-system-accept-default-p} is |
| 774 | non-@code{nil}, its value overrides the value of |
| 775 | @var{accept-default-p}. |
| 776 | @end defun |
| 777 | |
| 778 | Here are two functions you can use to let the user specify a coding |
| 779 | system, with completion. @xref{Completion}. |
| 780 | |
| 781 | @defun read-coding-system prompt &optional default |
| 782 | This function reads a coding system using the minibuffer, prompting with |
| 783 | string @var{prompt}, and returns the coding system name as a symbol. If |
| 784 | the user enters null input, @var{default} specifies which coding system |
| 785 | to return. It should be a symbol or a string. |
| 786 | @end defun |
| 787 | |
| 788 | @defun read-non-nil-coding-system prompt |
| 789 | This function reads a coding system using the minibuffer, prompting with |
| 790 | string @var{prompt}, and returns the coding system name as a symbol. If |
| 791 | the user tries to enter null input, it asks the user to try again. |
| 792 | @xref{Coding Systems}. |
| 793 | @end defun |
| 794 | |
| 795 | @node Default Coding Systems |
| 796 | @subsection Default Coding Systems |
| 797 | |
| 798 | This section describes variables that specify the default coding |
| 799 | system for certain files or when running certain subprograms, and the |
| 800 | function that I/O operations use to access them. |
| 801 | |
| 802 | The idea of these variables is that you set them once and for all to the |
| 803 | defaults you want, and then do not change them again. To specify a |
| 804 | particular coding system for a particular operation in a Lisp program, |
| 805 | don't change these variables; instead, override them using |
| 806 | @code{coding-system-for-read} and @code{coding-system-for-write} |
| 807 | (@pxref{Specifying Coding Systems}). |
| 808 | |
| 809 | @defvar auto-coding-regexp-alist |
| 810 | This variable is an alist of text patterns and corresponding coding |
| 811 | systems. Each element has the form @code{(@var{regexp} |
| 812 | . @var{coding-system})}; a file whose first few kilobytes match |
| 813 | @var{regexp} is decoded with @var{coding-system} when its contents are |
| 814 | read into a buffer. The settings in this alist take priority over |
| 815 | @code{coding:} tags in the files and the contents of |
| 816 | @code{file-coding-system-alist} (see below). The default value is set |
| 817 | so that Emacs automatically recognizes mail files in Babyl format and |
| 818 | reads them with no code conversions. |
| 819 | @end defvar |
| 820 | |
| 821 | @defvar file-coding-system-alist |
| 822 | This variable is an alist that specifies the coding systems to use for |
| 823 | reading and writing particular files. Each element has the form |
| 824 | @code{(@var{pattern} . @var{coding})}, where @var{pattern} is a regular |
| 825 | expression that matches certain file names. The element applies to file |
| 826 | names that match @var{pattern}. |
| 827 | |
| 828 | The @sc{cdr} of the element, @var{coding}, should be either a coding |
| 829 | system, a cons cell containing two coding systems, or a function name (a |
| 830 | symbol with a function definition). If @var{coding} is a coding system, |
| 831 | that coding system is used for both reading the file and writing it. If |
| 832 | @var{coding} is a cons cell containing two coding systems, its @sc{car} |
| 833 | specifies the coding system for decoding, and its @sc{cdr} specifies the |
| 834 | coding system for encoding. |
| 835 | |
| 836 | If @var{coding} is a function name, the function must return a coding |
| 837 | system or a cons cell containing two coding systems. This value is used |
| 838 | as described above. |
| 839 | @end defvar |
| 840 | |
| 841 | @defvar process-coding-system-alist |
| 842 | This variable is an alist specifying which coding systems to use for a |
| 843 | subprocess, depending on which program is running in the subprocess. It |
| 844 | works like @code{file-coding-system-alist}, except that @var{pattern} is |
| 845 | matched against the program name used to start the subprocess. The coding |
| 846 | system or systems specified in this alist are used to initialize the |
| 847 | coding systems used for I/O to the subprocess, but you can specify |
| 848 | other coding systems later using @code{set-process-coding-system}. |
| 849 | @end defvar |
| 850 | |
| 851 | @strong{Warning:} Coding systems such as @code{undecided}, which |
| 852 | determine the coding system from the data, do not work entirely reliably |
| 853 | with asynchronous subprocess output. This is because Emacs handles |
| 854 | asynchronous subprocess output in batches, as it arrives. If the coding |
| 855 | system leaves the character code conversion unspecified, or leaves the |
| 856 | end-of-line conversion unspecified, Emacs must try to detect the proper |
| 857 | conversion from one batch at a time, and this does not always work. |
| 858 | |
| 859 | Therefore, with an asynchronous subprocess, if at all possible, use a |
| 860 | coding system which determines both the character code conversion and |
| 861 | the end of line conversion---that is, one like @code{latin-1-unix}, |
| 862 | rather than @code{undecided} or @code{latin-1}. |
| 863 | |
| 864 | @defvar network-coding-system-alist |
| 865 | This variable is an alist that specifies the coding system to use for |
| 866 | network streams. It works much like @code{file-coding-system-alist}, |
| 867 | with the difference that the @var{pattern} in an element may be either a |
| 868 | port number or a regular expression. If it is a regular expression, it |
| 869 | is matched against the network service name used to open the network |
| 870 | stream. |
| 871 | @end defvar |
| 872 | |
| 873 | @defvar default-process-coding-system |
| 874 | This variable specifies the coding systems to use for subprocess (and |
| 875 | network stream) input and output, when nothing else specifies what to |
| 876 | do. |
| 877 | |
| 878 | The value should be a cons cell of the form @code{(@var{input-coding} |
| 879 | . @var{output-coding})}. Here @var{input-coding} applies to input from |
| 880 | the subprocess, and @var{output-coding} applies to output to it. |
| 881 | @end defvar |
| 882 | |
| 883 | @defun find-operation-coding-system operation &rest arguments |
| 884 | This function returns the coding system to use (by default) for |
| 885 | performing @var{operation} with @var{arguments}. The value has this |
| 886 | form: |
| 887 | |
| 888 | @example |
| 889 | (@var{decoding-system} @var{encoding-system}) |
| 890 | @end example |
| 891 | |
| 892 | The first element, @var{decoding-system}, is the coding system to use |
| 893 | for decoding (in case @var{operation} does decoding), and |
| 894 | @var{encoding-system} is the coding system for encoding (in case |
| 895 | @var{operation} does encoding). |
| 896 | |
| 897 | The argument @var{operation} should be a symbol, one of |
| 898 | @code{insert-file-contents}, @code{write-region}, @code{call-process}, |
| 899 | @code{call-process-region}, @code{start-process}, or |
| 900 | @code{open-network-stream}. These are the names of the Emacs I/O primitives |
| 901 | that can do coding system conversion. |
| 902 | |
| 903 | The remaining arguments should be the same arguments that might be given |
| 904 | to that I/O primitive. Depending on the primitive, one of those |
| 905 | arguments is selected as the @dfn{target}. For example, if |
| 906 | @var{operation} does file I/O, whichever argument specifies the file |
| 907 | name is the target. For subprocess primitives, the process name is the |
| 908 | target. For @code{open-network-stream}, the target is the service name |
| 909 | or port number. |
| 910 | |
| 911 | This function looks up the target in @code{file-coding-system-alist}, |
| 912 | @code{process-coding-system-alist}, or |
| 913 | @code{network-coding-system-alist}, depending on @var{operation}. |
| 914 | @xref{Default Coding Systems}. |
| 915 | @end defun |
| 916 | |
| 917 | @node Specifying Coding Systems |
| 918 | @subsection Specifying a Coding System for One Operation |
| 919 | |
| 920 | You can specify the coding system for a specific operation by binding |
| 921 | the variables @code{coding-system-for-read} and/or |
| 922 | @code{coding-system-for-write}. |
| 923 | |
| 924 | @defvar coding-system-for-read |
| 925 | If this variable is non-@code{nil}, it specifies the coding system to |
| 926 | use for reading a file, or for input from a synchronous subprocess. |
| 927 | |
| 928 | It also applies to any asynchronous subprocess or network stream, but in |
| 929 | a different way: the value of @code{coding-system-for-read} when you |
| 930 | start the subprocess or open the network stream specifies the input |
| 931 | decoding method for that subprocess or network stream. It remains in |
| 932 | use for that subprocess or network stream unless and until overridden. |
| 933 | |
| 934 | The right way to use this variable is to bind it with @code{let} for a |
| 935 | specific I/O operation. Its global value is normally @code{nil}, and |
| 936 | you should not globally set it to any other value. Here is an example |
| 937 | of the right way to use the variable: |
| 938 | |
| 939 | @example |
| 940 | ;; @r{Read the file with no character code conversion.} |
| 941 | ;; @r{Assume @sc{crlf} represents end-of-line.} |
| 942 | (let ((coding-system-for-write 'emacs-mule-dos)) |
| 943 | (insert-file-contents filename)) |
| 944 | @end example |
| 945 | |
| 946 | When its value is non-@code{nil}, @code{coding-system-for-read} takes |
| 947 | precedence over all other methods of specifying a coding system to use for |
| 948 | input, including @code{file-coding-system-alist}, |
| 949 | @code{process-coding-system-alist} and |
| 950 | @code{network-coding-system-alist}. |
| 951 | @end defvar |
| 952 | |
| 953 | @defvar coding-system-for-write |
| 954 | This works much like @code{coding-system-for-read}, except that it |
| 955 | applies to output rather than input. It affects writing to files, |
| 956 | as well as sending output to subprocesses and net connections. |
| 957 | |
| 958 | When a single operation does both input and output, as do |
| 959 | @code{call-process-region} and @code{start-process}, both |
| 960 | @code{coding-system-for-read} and @code{coding-system-for-write} |
| 961 | affect it. |
| 962 | @end defvar |
| 963 | |
| 964 | @defvar inhibit-eol-conversion |
| 965 | When this variable is non-@code{nil}, no end-of-line conversion is done, |
| 966 | no matter which coding system is specified. This applies to all the |
| 967 | Emacs I/O and subprocess primitives, and to the explicit encoding and |
| 968 | decoding functions (@pxref{Explicit Encoding}). |
| 969 | @end defvar |
| 970 | |
| 971 | @node Explicit Encoding |
| 972 | @subsection Explicit Encoding and Decoding |
| 973 | @cindex encoding text |
| 974 | @cindex decoding text |
| 975 | |
| 976 | All the operations that transfer text in and out of Emacs have the |
| 977 | ability to use a coding system to encode or decode the text. |
| 978 | You can also explicitly encode and decode text using the functions |
| 979 | in this section. |
| 980 | |
| 981 | The result of encoding, and the input to decoding, are not ordinary |
| 982 | text. They logically consist of a series of byte values; that is, a |
| 983 | series of characters whose codes are in the range 0 through 255. In a |
| 984 | multibyte buffer or string, character codes 128 through 159 are |
| 985 | represented by multibyte sequences, but this is invisible to Lisp |
| 986 | programs. |
| 987 | |
| 988 | The usual way to read a file into a buffer as a sequence of bytes, so |
| 989 | you can decode the contents explicitly, is with |
| 990 | @code{insert-file-contents-literally} (@pxref{Reading from Files}); |
| 991 | alternatively, specify a non-@code{nil} @var{rawfile} argument when |
| 992 | visiting a file with @code{find-file-noselect}. These methods result in |
| 993 | a unibyte buffer. |
| 994 | |
| 995 | The usual way to use the byte sequence that results from explicitly |
| 996 | encoding text is to copy it to a file or process---for example, to write |
| 997 | it with @code{write-region} (@pxref{Writing to Files}), and suppress |
| 998 | encoding by binding @code{coding-system-for-write} to |
| 999 | @code{no-conversion}. |
| 1000 | |
| 1001 | Here are the functions to perform explicit encoding or decoding. The |
| 1002 | decoding functions produce sequences of bytes; the encoding functions |
| 1003 | are meant to operate on sequences of bytes. All of these functions |
| 1004 | discard text properties. |
| 1005 | |
| 1006 | @defun encode-coding-region start end coding-system |
| 1007 | This function encodes the text from @var{start} to @var{end} according |
| 1008 | to coding system @var{coding-system}. The encoded text replaces the |
| 1009 | original text in the buffer. The result of encoding is logically a |
| 1010 | sequence of bytes, but the buffer remains multibyte if it was multibyte |
| 1011 | before. |
| 1012 | @end defun |
| 1013 | |
| 1014 | @defun encode-coding-string string coding-system |
| 1015 | This function encodes the text in @var{string} according to coding |
| 1016 | system @var{coding-system}. It returns a new string containing the |
| 1017 | encoded text. The result of encoding is a unibyte string. |
| 1018 | @end defun |
| 1019 | |
| 1020 | @defun decode-coding-region start end coding-system |
| 1021 | This function decodes the text from @var{start} to @var{end} according |
| 1022 | to coding system @var{coding-system}. The decoded text replaces the |
| 1023 | original text in the buffer. To make explicit decoding useful, the text |
| 1024 | before decoding ought to be a sequence of byte values, but both |
| 1025 | multibyte and unibyte buffers are acceptable. |
| 1026 | @end defun |
| 1027 | |
| 1028 | @defun decode-coding-string string coding-system |
| 1029 | This function decodes the text in @var{string} according to coding |
| 1030 | system @var{coding-system}. It returns a new string containing the |
| 1031 | decoded text. To make explicit decoding useful, the contents of |
| 1032 | @var{string} ought to be a sequence of byte values, but a multibyte |
| 1033 | string is acceptable. |
| 1034 | @end defun |
| 1035 | |
| 1036 | @node Terminal I/O Encoding |
| 1037 | @subsection Terminal I/O Encoding |
| 1038 | |
| 1039 | Emacs can decode keyboard input using a coding system, and encode |
| 1040 | terminal output. This is useful for terminals that transmit or display |
| 1041 | text using a particular encoding such as Latin-1. Emacs does not set |
| 1042 | @code{last-coding-system-used} for encoding or decoding for the |
| 1043 | terminal. |
| 1044 | |
| 1045 | @defun keyboard-coding-system |
| 1046 | This function returns the coding system that is in use for decoding |
| 1047 | keyboard input---or @code{nil} if no coding system is to be used. |
| 1048 | @end defun |
| 1049 | |
| 1050 | @defun set-keyboard-coding-system coding-system |
| 1051 | This function specifies @var{coding-system} as the coding system to |
| 1052 | use for decoding keyboard input. If @var{coding-system} is @code{nil}, |
| 1053 | that means do not decode keyboard input. |
| 1054 | @end defun |
| 1055 | |
| 1056 | @defun terminal-coding-system |
| 1057 | This function returns the coding system that is in use for encoding |
| 1058 | terminal output---or @code{nil} for no encoding. |
| 1059 | @end defun |
| 1060 | |
| 1061 | @defun set-terminal-coding-system coding-system |
| 1062 | This function specifies @var{coding-system} as the coding system to use |
| 1063 | for encoding terminal output. If @var{coding-system} is @code{nil}, |
| 1064 | that means do not encode terminal output. |
| 1065 | @end defun |
| 1066 | |
| 1067 | @node MS-DOS File Types |
| 1068 | @subsection MS-DOS File Types |
| 1069 | @cindex DOS file types |
| 1070 | @cindex MS-DOS file types |
| 1071 | @cindex Windows file types |
| 1072 | @cindex file types on MS-DOS and Windows |
| 1073 | @cindex text files and binary files |
| 1074 | @cindex binary files and text files |
| 1075 | |
| 1076 | On MS-DOS and Microsoft Windows, Emacs guesses the appropriate |
| 1077 | end-of-line conversion for a file by looking at the file's name. This |
| 1078 | feature classifies files as @dfn{text files} and @dfn{binary files}. By |
| 1079 | ``binary file'' we mean a file of literal byte values that are not |
| 1080 | necessarily meant to be characters; Emacs does no end-of-line conversion |
| 1081 | and no character code conversion for them. On the other hand, the bytes |
| 1082 | in a text file are intended to represent characters; when you create a |
| 1083 | new file whose name implies that it is a text file, Emacs uses DOS |
| 1084 | end-of-line conversion. |
| 1085 | |
| 1086 | @defvar buffer-file-type |
| 1087 | This variable, automatically buffer-local in each buffer, records the |
| 1088 | file type of the buffer's visited file. When a buffer does not specify |
| 1089 | a coding system with @code{buffer-file-coding-system}, this variable is |
| 1090 | used to determine which coding system to use when writing the contents |
| 1091 | of the buffer. It should be @code{nil} for text, @code{t} for binary. |
| 1092 | If it is @code{t}, the coding system is @code{no-conversion}. |
| 1093 | Otherwise, @code{undecided-dos} is used. |
| 1094 | |
| 1095 | Normally this variable is set by visiting a file; it is set to |
| 1096 | @code{nil} if the file was visited without any actual conversion. |
| 1097 | @end defvar |
| 1098 | |
| 1099 | @defopt file-name-buffer-file-type-alist |
| 1100 | This variable holds an alist for recognizing text and binary files. |
| 1101 | Each element has the form (@var{regexp} . @var{type}), where |
| 1102 | @var{regexp} is matched against the file name, and @var{type} may be |
| 1103 | @code{nil} for text, @code{t} for binary, or a function to call to |
| 1104 | compute which. If it is a function, then it is called with a single |
| 1105 | argument (the file name) and should return @code{t} or @code{nil}. |
| 1106 | |
| 1107 | When running on MS-DOS or MS-Windows, Emacs checks this alist to decide |
| 1108 | which coding system to use when reading a file. For a text file, |
| 1109 | @code{undecided-dos} is used. For a binary file, @code{no-conversion} |
| 1110 | is used. |
| 1111 | |
| 1112 | If no element in this alist matches a given file name, then |
| 1113 | @code{default-buffer-file-type} says how to treat the file. |
| 1114 | @end defopt |
| 1115 | |
| 1116 | @defopt default-buffer-file-type |
| 1117 | This variable says how to handle files for which |
| 1118 | @code{file-name-buffer-file-type-alist} says nothing about the type. |
| 1119 | |
| 1120 | If this variable is non-@code{nil}, then these files are treated as |
| 1121 | binary: the coding system @code{no-conversion} is used. Otherwise, |
| 1122 | nothing special is done for them---the coding system is deduced solely |
| 1123 | from the file contents, in the usual Emacs fashion. |
| 1124 | @end defopt |
| 1125 | |
| 1126 | @node Input Methods |
| 1127 | @section Input Methods |
| 1128 | @cindex input methods |
| 1129 | |
| 1130 | @dfn{Input methods} provide convenient ways of entering non-@sc{ascii} |
| 1131 | characters from the keyboard. Unlike coding systems, which translate |
| 1132 | non-@sc{ascii} characters to and from encodings meant to be read by |
| 1133 | programs, input methods provide human-friendly commands. (@xref{Input |
| 1134 | Methods,,, emacs, The GNU Emacs Manual}, for information on how users |
| 1135 | use input methods to enter text.) How to define input methods is not |
| 1136 | yet documented in this manual, but here we describe how to use them. |
| 1137 | |
| 1138 | Each input method has a name, which is currently a string; |
| 1139 | in the future, symbols may also be usable as input method names. |
| 1140 | |
| 1141 | @defvar current-input-method |
| 1142 | This variable holds the name of the input method now active in the |
| 1143 | current buffer. (It automatically becomes local in each buffer when set |
| 1144 | in any fashion.) It is @code{nil} if no input method is active in the |
| 1145 | buffer now. |
| 1146 | @end defvar |
| 1147 | |
| 1148 | @defvar default-input-method |
| 1149 | This variable holds the default input method for commands that choose an |
| 1150 | input method. Unlike @code{current-input-method}, this variable is |
| 1151 | normally global. |
| 1152 | @end defvar |
| 1153 | |
| 1154 | @defun set-input-method input-method |
| 1155 | This function activates input method @var{input-method} for the current |
| 1156 | buffer. It also sets @code{default-input-method} to @var{input-method}. |
| 1157 | If @var{input-method} is @code{nil}, this function deactivates any input |
| 1158 | method for the current buffer. |
| 1159 | @end defun |
| 1160 | |
| 1161 | @defun read-input-method-name prompt &optional default inhibit-null |
| 1162 | This function reads an input method name with the minibuffer, prompting |
| 1163 | with @var{prompt}. If @var{default} is non-@code{nil}, that is returned |
| 1164 | by default, if the user enters empty input. However, if |
| 1165 | @var{inhibit-null} is non-@code{nil}, empty input signals an error. |
| 1166 | |
| 1167 | The returned value is a string. |
| 1168 | @end defun |
| 1169 | |
| 1170 | @defvar input-method-alist |
| 1171 | This variable defines all the supported input methods. |
| 1172 | Each element defines one input method, and should have the form: |
| 1173 | |
| 1174 | @example |
| 1175 | (@var{input-method} @var{language-env} @var{activate-func} |
| 1176 | @var{title} @var{description} @var{args}...) |
| 1177 | @end example |
| 1178 | |
| 1179 | Here @var{input-method} is the input method name, a string; |
| 1180 | @var{language-env} is another string, the name of the language |
| 1181 | environment this input method is recommended for. (That serves only for |
| 1182 | documentation purposes.) |
| 1183 | |
| 1184 | @var{activate-func} is a function to call to activate this method. The |
| 1185 | @var{args}, if any, are passed as arguments to @var{activate-func}. All |
| 1186 | told, the arguments to @var{activate-func} are @var{input-method} and |
| 1187 | the @var{args}. |
| 1188 | |
| 1189 | @var{title} is a string to display in the mode line while this method is |
| 1190 | active. @var{description} is a string describing this method and what |
| 1191 | it is good for. |
| 1192 | @end defvar |
| 1193 | |
| 1194 | The fundamental interface to input methods is through the |
| 1195 | variable @code{input-method-function}. @xref{Reading One Event}. |
| 1196 | |
| 1197 | @node Locales |
| 1198 | @section Locales |
| 1199 | @cindex locale |
| 1200 | |
| 1201 | POSIX defines a concept of ``locales'' which control which language |
| 1202 | to use in language-related features. These Emacs variables control |
| 1203 | how Emacs interacts with these features. |
| 1204 | |
| 1205 | @defvar locale-coding-system |
| 1206 | @tindex locale-coding-system |
| 1207 | This variable specifies the coding system to use for decoding system |
| 1208 | error messages, for encoding the format argument to |
| 1209 | @code{format-time-string}, and for decoding the return value of |
| 1210 | @code{format-time-string}. |
| 1211 | @end defvar |
| 1212 | |
| 1213 | @defvar system-messages-locale |
| 1214 | @tindex system-messages-locale |
| 1215 | This variable specifies the locale to use for generating system error |
| 1216 | messages. Changing the locale can cause messages to come out in a |
| 1217 | different language or in a different orthography. If the variable is |
| 1218 | @code{nil}, the locale is specified by environment variables in the |
| 1219 | usual POSIX fashion. |
| 1220 | @end defvar |
| 1221 | |
| 1222 | @defvar system-time-locale |
| 1223 | @tindex system-time-locale |
| 1224 | This variable specifies the locale to use for formatting time values. |
| 1225 | Changing the locale can cause messages to appear according to the |
| 1226 | conventions of a different language. If the variable is @code{nil}, the |
| 1227 | locale is specified by environment variables in the usual POSIX fashion. |
| 1228 | @end defvar |
| 1229 | |