characters and how they are stored in strings and buffers.
@menu
-* Text Representations::
-* Converting Representations::
-* Selecting a Representation::
-* Character Codes::
-* Character Sets::
-* Chars and Bytes::
-* Splitting Characters::
-* Scanning Charsets::
-* Translation of Characters::
-* Coding Systems::
-* Input Methods::
-* Locales:: Interacting with the POSIX locale.
+* Text Representations:: Unibyte and multibyte representations
+* Converting Representations:: Converting unibyte to multibyte and vice versa.
+* Selecting a Representation:: Treating a byte sequence as unibyte or multi.
+* Character Codes:: How unibyte and multibyte relate to
+ codes of individual characters.
+* Character Sets:: The space of possible characters codes
+ is divided into various character sets.
+* Chars and Bytes:: More information about multibyte encodings.
+* Splitting Characters:: Converting a character to its byte sequence.
+* Scanning Charsets:: Which character sets are used in a buffer?
+* Translation of Characters:: Translation tables are used for conversion.
+* Coding Systems:: Coding systems are conversions for saving files.
+* Input Methods:: Input methods allow users to enter various
+ non-ASCII characters without special keyboards.
+* Locales:: Interacting with the POSIX locale.
@end menu
@node Text Representations
character are always in the range 160 through 255 (octal 0240 through
0377); these values are @dfn{trailing codes}.
- Some sequences of bytes do not form meaningful multibyte characters:
-for example, a single isolated byte in the range 128 through 255 is
-never meaningful. Such byte sequences are not entirely valid, and never
-appear in proper multibyte text (since that consists of a sequence of
-@emph{characters}); but they can appear as part of ``raw bytes''
-(@pxref{Explicit Encoding}).
+ Some sequences of bytes are not valid in multibyte text: for example,
+a single isolated byte in the range 128 through 159 is not allowed. But
+character codes 128 through 159 can appear in multibyte text,
+represented as two-byte sequences. All the character codes 128 through
+255 are possible (though slightly abnormal) in multibyte text; they
+appear in multibyte buffers and strings when you do explicit encoding
+and decoding (@pxref{Explicit Encoding}).
In a buffer, the buffer-local value of the variable
@code{enable-multibyte-characters} specifies the representation used.
user that cannot be overridden automatically.
Converting unibyte text to multibyte text leaves @sc{ascii} characters
-unchanged, and likewise 128 through 159. It converts the non-@sc{ascii}
-codes 160 through 255 by adding the value @code{nonascii-insert-offset}
-to each character code. By setting this variable, you specify which
-character set the unibyte characters correspond to (@pxref{Character
-Sets}). For example, if @code{nonascii-insert-offset} is 2048, which is
-@code{(- (make-char 'latin-iso8859-1) 128)}, then the unibyte
-non-@sc{ascii} characters correspond to Latin 1. If it is 2688, which
-is @code{(- (make-char 'greek-iso8859-7) 128)}, then they correspond to
-Greek letters.
+unchanged, and likewise character codes 128 through 159. It converts
+the non-@sc{ascii} codes 160 through 255 by adding the value
+@code{nonascii-insert-offset} to each character code. By setting this
+variable, you specify which character set the unibyte characters
+correspond to (@pxref{Character Sets}). For example, if
+@code{nonascii-insert-offset} is 2048, which is @code{(- (make-char
+'latin-iso8859-1) 128)}, then the unibyte non-@sc{ascii} characters
+correspond to Latin 1. If it is 2688, which is @code{(- (make-char
+'greek-iso8859-7) 128)}, then they correspond to Greek letters.
Converting multibyte text to unibyte is simpler: it discards all but
the low 8 bits of each character code. If @code{nonascii-insert-offset}
This variable specifies the amount to add to a non-@sc{ascii} character
when converting unibyte text to multibyte. It also applies when
@code{self-insert-command} inserts a character in the unibyte
-non-@sc{ascii} range, 128 through 255. However, the function
-@code{insert-char} does not perform this conversion.
+non-@sc{ascii} range, 128 through 255. However, the functions
+@code{insert} and @code{insert-char} do not perform this conversion.
The right value to use to select character set @var{cs} is @code{(-
(make-char @var{cs}) 128)}. If the value of
This variable provides a more general alternative to
@code{nonascii-insert-offset}. You can use it to specify independently
how to translate each code in the range of 128 through 255 into a
-multibyte character. The value should be a vector, or @code{nil}.
+multibyte character. The value should be a char-table, or @code{nil}.
If this is non-@code{nil}, it overrides @code{nonascii-insert-offset}.
@end defvar
This function converts the text of @var{string} to unibyte
representation, if it isn't already, and returns the result. If
@var{string} is a unibyte string, it is returned unchanged.
+Multibyte character codes are converted to unibyte
+by using just the low 8 bits.
@end defun
@defun string-make-multibyte string
This function converts the text of @var{string} to multibyte
representation, if it isn't already, and returns the result. If
@var{string} is a multibyte string, it is returned unchanged.
+The function @code{unibyte-char-to-multibyte} is used to convert
+each unibyte character to a multibyte character.
@end defun
@node Selecting a Representation
sequence of bytes. As a consequence, it can change the contents viewed
as characters; a sequence of two bytes which is treated as one character
in multibyte representation will count as two characters in unibyte
-representation.
+representation. Character codes 128 through 159 are an exception. They
+are represented by one byte in a unibyte buffer, but when the buffer is
+set to multibyte, they are converted to two-byte sequences, and vice
+versa.
This function sets @code{enable-multibyte-characters} to record which
representation is in use. It also adjusts various data in the buffer
more characters than @var{string} has.
If @var{string} is already a unibyte string, then the value is
-@var{string} itself.
+@var{string} itself. Otherwise it is a newly created string, with no
+text properties. If @var{string} is multibyte, any characters it
+contains of charset @var{eight-bit-control} or @var{eight-bit-graphic}
+are converted to the corresponding single byte.
@end defun
@defun string-as-multibyte string
value may have fewer characters than @var{string} has.
If @var{string} is already a multibyte string, then the value is
-@var{string} itself.
+@var{string} itself. Otherwise it is a newly created string, with no
+text properties. If @var{string} is unibyte and contains any individual
+8-bit bytes (i.e.@: not part of a multibyte form), they are converted to
+the corresponding multibyte character of charset @var{eight-bit-control}
+or @var{eight-bit-graphic}.
@end defun
@node Character Codes
codes. The valid character codes for unibyte representation range from
0 to 255---the values that can fit in one byte. The valid character
codes for multibyte representation range from 0 to 524287, but not all
-values in that range are valid. In particular, the values 128 through
-255 are not legitimate in multibyte text (though they can occur in ``raw
-bytes''; @pxref{Explicit Encoding}). Only the @sc{ascii} codes 0
-through 127 are fully legitimate in both representations.
+values in that range are valid. The values 128 through 255 are not
+entirely proper in multibyte text, but they can occur if you do explicit
+encoding and decoding (@pxref{Explicit Encoding}). Some other character
+codes cannot occur at all in multibyte text. Only the @sc{ascii} codes
+0 through 127 are completely legitimate in both representations.
-@defun char-valid-p charcode
+@defun char-valid-p charcode &optional genericp
This returns @code{t} if @var{charcode} is valid for either one of the two
text representations.
(char-valid-p 2248)
@result{} t
@end example
+
+If the optional argument @var{genericp} is non-nil, this function
+returns @code{t} if @var{charcode} is a generic character
+(@pxref{Splitting Characters}).
@end defun
@node Character Sets
characters, generally known as Big 5, is divided into two Emacs
character sets, @code{chinese-big5-1} and @code{chinese-big5-2}.
+ @sc{ascii} characters are in character set @code{ascii}. The
+non-@sc{ascii} characters 128 through 159 are in character set
+@code{eight-bit-control}, and codes 160 through 255 are in character set
+@code{eight-bit-graphic}.
+
@defun charsetp object
Returns @code{t} if @var{object} is a symbol that names a character set,
@code{nil} otherwise.
This function returns the charset property list of the character set
@var{charset}. Although @var{charset} is a symbol, this is not the same
as the property list of that symbol. Charset properties are used for
-special purposes within Emacs; for example, @code{x-charset-registry}
-helps determine which fonts to use (@pxref{Font Selection}).
+special purposes within Emacs; for example,
+@code{preferred-coding-system} helps determine which coding system to
+use to encode characters in a charset.
@end defun
@node Chars and Bytes
In multibyte representation, each character occupies one or more
bytes. Each character set has an @dfn{introduction sequence}, which is
normally one or two bytes long. (Exception: the @sc{ascii} character
-set has a zero-length introduction sequence.) The introduction sequence
-is the beginning of the byte sequence for any character in the character
-set. The rest of the character's bytes distinguish it from the other
-characters in the same character set. Depending on the character set,
-there are either one or two distinguishing bytes; the number of such
-bytes is called the @dfn{dimension} of the character set.
+set and the @sc{eight-bit-graphic} character set have a zero-length
+introduction sequence.) The introduction sequence is the beginning of
+the byte sequence for any character in the character set. The rest of
+the character's bytes distinguish it from the other characters in the
+same character set. Depending on the character set, there are either
+one or two distinguishing bytes; the number of such bytes is called the
+@dfn{dimension} of the character set.
@defun charset-dimension charset
This function returns the dimension of @var{charset}; at present, the
@result{} (latin-iso8859-1 72)
(split-char 65)
@result{} (ascii 65)
-@end example
-
-Unibyte non-@sc{ascii} characters are considered as part of
-the @code{ascii} character set:
-
-@example
-(split-char 192)
- @result{} (ascii 192)
+(split-char 128)
+ @result{} (eight-bit-control 128)
@end example
@end defun
-@defun make-char charset &rest byte-values
-This function returns the character in character set @var{charset}
-identified by @var{byte-values}. This is roughly the inverse of
-@code{split-char}. Normally, you should specify either one or two
-@var{byte-values}, according to the dimension of @var{charset}. For
-example,
+@defun make-char charset &optional code1 code2
+This function returns the character in character set @var{charset} whose
+position codes are @var{code1} and @var{code2}. This is roughly the
+inverse of @code{split-char}. Normally, you should specify either one
+or both of @var{code1} and @var{code2} according to the dimension of
+@var{charset}. For example,
@example
(make-char 'latin-iso8859-1 72)
@result{} 2176
(char-valid-p 2176)
@result{} nil
+(char-valid-p 2176 t)
+ @result{} t
(split-char 2176)
@result{} (latin-iso8859-1 0)
@end example
+The character sets @sc{ascii}, @sc{eight-bit-control}, and
+@sc{eight-bit-graphic} don't have corresponding generic characters. If
+@var{charset} is one of them and you don't supply @var{code1},
+@code{make-char} returns the character code corresponding to the
+smallest code in @var{charset}.
+
@node Scanning Charsets
@section Scanning for Character Sets
is non-@code{nil}, then each character in the region is translated
through this table, and the value returned describes the translated
characters instead of the characters actually in the buffer.
-
-In two peculiar cases, the value includes the symbol @code{unknown}:
-
-@itemize @bullet
-@item
-When a unibyte buffer contains non-@sc{ascii} characters.
-
-@item
-When a multibyte buffer contains invalid byte-sequences (raw bytes).
-@xref{Explicit Encoding}.
-@end itemize
@end defun
@defun find-charset-string string &optional translation
@defun make-translation-table &rest translations
This function returns a translation table based on the argument
-@var{translations}. Each element of
-@var{translations} should be a list of the form @code{(@var{from}
-. @var{to})}; this says to translate the character @var{from} into
-@var{to}.
+@var{translations}. Each element of @var{translations} should be a
+list of elements of the form @code{(@var{from} . @var{to})}; this says
+to translate the character @var{from} into @var{to}.
+
+The arguments and the forms in each argument are processed in order,
+and if a previous form already translates @var{to} to some other
+character, say @var{to-alt}, @var{from} is also translated to
+@var{to-alt}.
You can also map one whole character set into another character set with
the same dimension. To do this, you specify a generic character (which
documented here.
@menu
-* Coding System Basics::
-* Encoding and I/O::
-* Lisp and Coding Systems::
-* User-Chosen Coding Systems::
-* Default Coding Systems::
-* Specifying Coding Systems::
-* Explicit Encoding::
-* Terminal I/O Encoding::
-* MS-DOS File Types::
+* Coding System Basics:: Basic concepts.
+* Encoding and I/O:: How file I/O functions handle coding systems.
+* Lisp and Coding Systems:: Functions to operate on coding system names.
+* User-Chosen Coding Systems:: Asking the user to choose a coding system.
+* Default Coding Systems:: Controlling the default choices.
+* Specifying Coding Systems:: Requesting a particular coding system
+ for a single file operation.
+* Explicit Encoding:: Encoding or decoding text without doing I/O.
+* Terminal I/O Encoding:: Use of encoding for terminal I/O.
+* MS-DOS File Types:: How DOS "text" and "binary" files
+ relate to coding systems.
@end menu
@node Coding System Basics
@end defvar
@defvar save-buffer-coding-system
-This variable specifies the coding system for saving the buffer---but it
-is not used for @code{write-region}.
+This variable specifies the coding system for saving the buffer (by
+overriding @code{buffer-file-coding-system}). Note that it is not used
+for @code{write-region}.
When a command to save the buffer starts out to use
-@code{save-buffer-coding-system}, and that coding system cannot handle
+@code{buffer-file-coding-system} (or @code{save-buffer-coding-system}),
+and that coding system cannot handle
the actual text in the buffer, the command asks the user to choose
another coding system. After that happens, the command also updates
-@code{save-buffer-coding-system} to represent the coding system that the
+@code{buffer-file-coding-system} to represent the coding system that the
user specified.
@end defvar
@defun coding-system-list &optional base-only
This function returns a list of all coding system names (symbols). If
@var{base-only} is non-@code{nil}, the value includes only the
-base coding systems. Otherwise, it includes variant coding systems as well.
+base coding systems. Otherwise, it includes alias and variant coding
+systems as well.
@end defun
@defun coding-system-p object
@defun detect-coding-region start end &optional highest
This function chooses a plausible coding system for decoding the text
-from @var{start} to @var{end}. This text should be ``raw bytes''
+from @var{start} to @var{end}. This text should be a byte sequence
(@pxref{Explicit Encoding}).
Normally this function returns a list of coding systems that could
@node User-Chosen Coding Systems
@subsection User-Chosen Coding Systems
-@defun select-safe-coding-system from to &optional preferred-coding-system
-This function selects a coding system for encoding the text between
-@var{from} and @var{to}, asking the user to choose if necessary.
-
-The optional argument @var{preferred-coding-system} specifies a coding
-system to try first. If that one can handle the text in the specified
-region, then it is used. If this argument is omitted, the current
-buffer's value of @code{buffer-file-coding-system} is tried first.
-
-If the region contains some multibyte characters that the preferred
-coding system cannot encode, this function asks the user to choose from
-a list of coding systems which can encode the text, and returns the
-user's choice.
-
-One other kludgy feature: if @var{from} is a string, the string is the
-target text, and @var{to} is ignored.
+@defun select-safe-coding-system from to &optional default-coding-system accept-default-p
+This function selects a coding system for encoding specified text,
+asking the user to choose if necessary. Normally the specified text
+is the text in the current buffer between @var{from} and @var{to},
+defaulting to the whole buffer if they are @code{nil}. If @var{from}
+is a string, the string specifies the text to encode, and @var{to} is
+ignored.
+
+If @var{default-coding-system} is non-@code{nil}, that is the first
+coding system to try; if that can handle the text,
+@code{select-safe-coding-system} returns that coding system. It can
+also be a list of coding systems; then the function tries each of them
+one by one. After trying all of them, it next tries the user's most
+preferred coding system (@pxref{Recognize Coding,
+prefer-coding-system, the description of @code{prefer-coding-system},
+emacs, GNU Emacs Manual}), and after that the current buffer's value
+of @code{buffer-file-coding-system} (if it is not @code{undecided}).
+
+If one of those coding systems can safely encode all the specified
+text, @code{select-safe-coding-system} chooses it and returns it.
+Otherwise, it asks the user to choose from a list of coding systems
+which can encode all the text, and returns the user's choice.
+
+The optional argument @var{accept-default-p}, if non-@code{nil},
+should be a function to determine whether the coding system selected
+without user interaction is acceptable. If this function returns
+@code{nil}, the silently selected coding system is rejected, and the
+user is asked to select a coding system from a list of possible
+candidates.
+
+@vindex select-safe-coding-system-accept-default-p
+If the variable @code{select-safe-coding-system-accept-default-p} is
+non-@code{nil}, its value overrides the value of
+@var{accept-default-p}.
@end defun
Here are two functions you can use to let the user specify a coding
@code{coding-system-for-read} and @code{coding-system-for-write}
(@pxref{Specifying Coding Systems}).
+@defvar auto-coding-regexp-alist
+This variable is an alist of text patterns and corresponding coding
+systems. Each element has the form @code{(@var{regexp}
+. @var{coding-system})}; a file whose first few kilobytes match
+@var{regexp} is decoded with @var{coding-system} when its contents are
+read into a buffer. The settings in this alist take priority over
+@code{coding:} tags in the files and the contents of
+@code{file-coding-system-alist} (see below). The default value is set
+so that Emacs automatically recognizes mail files in Babyl format and
+reads them with no code conversions.
+@end defvar
+
@defvar file-coding-system-alist
This variable is an alist that specifies the coding systems to use for
reading and writing particular files. Each element has the form
You can also explicitly encode and decode text using the functions
in this section.
-@cindex raw bytes
The result of encoding, and the input to decoding, are not ordinary
-text. They are ``raw bytes''---bytes that represent text in the same
-way that an external file would. When a buffer contains raw bytes, it
-is most natural to mark that buffer as using unibyte representation,
-using @code{set-buffer-multibyte} (@pxref{Selecting a Representation}),
-but this is not required. If the buffer's contents are only temporarily
-raw, leave the buffer multibyte, which will be correct after you decode
-them.
-
- The usual way to get raw bytes in a buffer, for explicit decoding, is
-to read them from a file with @code{insert-file-contents-literally}
-(@pxref{Reading from Files}) or specify a non-@code{nil} @var{rawfile}
-argument when visiting a file with @code{find-file-noselect}.
-
- The usual way to use the raw bytes that result from explicitly
-encoding text is to copy them to a file or process---for example, to
-write them with @code{write-region} (@pxref{Writing to Files}), and
-suppress encoding for that @code{write-region} call by binding
-@code{coding-system-for-write} to @code{no-conversion}.
-
- Raw bytes typically contain stray individual bytes with values in the
-range 128 through 255, that are legitimate only as part of multibyte
-sequences. Even if the buffer is multibyte, Emacs treats each such
-individual byte as a character and uses the byte value as its character
-code. In this way, character codes 128 through 255 can be found in a
-multibyte buffer, even though they are not legitimate multibyte
-character codes.
-
- Raw bytes sometimes contain overlong byte-sequences that look like a
-proper multibyte character plus extra superfluous trailing codes. For
-most purposes, Emacs treats such a sequence in a buffer or string as a
-single character, and if you look at its character code, you get the
-value that corresponds to the multibyte character
-sequence---disregarding the extra trailing codes. This is not quite
-clean, but raw bytes are used only in limited ways, so as a practical
-matter it is not worth the trouble to treat this case differently.
-
- When a multibyte buffer contains illegitimate byte sequences,
-sometimes insertion or deletion can cause them to coalesce into a
-legitimate multibyte character. For example, suppose the buffer
-contains the sequence 129 68 192, 68 being the character @samp{D}. If
-you delete the @samp{D}, the bytes 129 and 192 become adjacent, and thus
-become one multibyte character (Latin-1 A with grave accent). Point
-moves to one side or the other of the character, since it cannot be
-within a character. Don't be alarmed by this.
-
- Some really peculiar situations prevent proper coalescence. For
-example, if you narrow the buffer so that the accessible portion begins
-just before the @samp{D}, then delete the @samp{D}, the two surrounding
-bytes cannot coalesce because one of them is outside the accessible
-portion of the buffer. In this case, the deletion cannot be done, so
-@code{delete-region} signals an error.
+text. They logically consist of a series of byte values; that is, a
+series of characters whose codes are in the range 0 through 255. In a
+multibyte buffer or string, character codes 128 through 159 are
+represented by multibyte sequences, but this is invisible to Lisp
+programs.
+
+ The usual way to read a file into a buffer as a sequence of bytes, so
+you can decode the contents explicitly, is with
+@code{insert-file-contents-literally} (@pxref{Reading from Files});
+alternatively, specify a non-@code{nil} @var{rawfile} argument when
+visiting a file with @code{find-file-noselect}. These methods result in
+a unibyte buffer.
+
+ The usual way to use the byte sequence that results from explicitly
+encoding text is to copy it to a file or process---for example, to write
+it with @code{write-region} (@pxref{Writing to Files}), and suppress
+encoding by binding @code{coding-system-for-write} to
+@code{no-conversion}.
Here are the functions to perform explicit encoding or decoding. The
-decoding functions produce ``raw bytes''; the encoding functions are
-meant to operate on ``raw bytes''. All of these functions discard text
-properties.
+decoding functions produce sequences of bytes; the encoding functions
+are meant to operate on sequences of bytes. All of these functions
+discard text properties.
@defun encode-coding-region start end coding-system
This function encodes the text from @var{start} to @var{end} according
to coding system @var{coding-system}. The encoded text replaces the
-original text in the buffer. The result of encoding is ``raw bytes,''
-but the buffer remains multibyte if it was multibyte before.
+original text in the buffer. The result of encoding is logically a
+sequence of bytes, but the buffer remains multibyte if it was multibyte
+before.
@end defun
@defun encode-coding-string string coding-system
This function encodes the text in @var{string} according to coding
system @var{coding-system}. It returns a new string containing the
-encoded text. The result of encoding is a unibyte string of ``raw bytes.''
+encoded text. The result of encoding is a unibyte string.
@end defun
@defun decode-coding-region start end coding-system
This function decodes the text from @var{start} to @var{end} according
to coding system @var{coding-system}. The decoded text replaces the
original text in the buffer. To make explicit decoding useful, the text
-before decoding ought to be ``raw bytes.''
+before decoding ought to be a sequence of byte values, but both
+multibyte and unibyte buffers are acceptable.
@end defun
@defun decode-coding-string string coding-system
This function decodes the text in @var{string} according to coding
system @var{coding-system}. It returns a new string containing the
decoded text. To make explicit decoding useful, the contents of
-@var{string} ought to be ``raw bytes.''
+@var{string} ought to be a sequence of byte values, but a multibyte
+string is acceptable.
@end defun
@node Terminal I/O Encoding
On MS-DOS and Microsoft Windows, Emacs guesses the appropriate
end-of-line conversion for a file by looking at the file's name. This
-feature classifies fils as @dfn{text files} and @dfn{binary files}. By
+feature classifies files as @dfn{text files} and @dfn{binary files}. By
``binary file'' we mean a file of literal byte values that are not
necessarily meant to be characters; Emacs does no end-of-line conversion
and no character code conversion for them. On the other hand, the bytes
environment this input method is recommended for. (That serves only for
documentation purposes.)
-@var{title} is a string to display in the mode line while this method is
-active. @var{description} is a string describing this method and what
-it is good for.
-
@var{activate-func} is a function to call to activate this method. The
@var{args}, if any, are passed as arguments to @var{activate-func}. All
told, the arguments to @var{activate-func} are @var{input-method} and
the @var{args}.
+
+@var{title} is a string to display in the mode line while this method is
+active. @var{description} is a string describing this method and what
+it is good for.
@end defvar
The fundamental interface to input methods is through the
conventions of a different language. If the variable is @code{nil}, the
locale is specified by environment variables in the usual POSIX fashion.
@end defvar
+
HCoop / bpt / emacs.git / blobdiff