New directory

[bpt/emacs.git] / lispref / nonascii.texi

@c -*-texinfo-*-
@c This is part of the GNU Emacs Lisp Reference Manual.
@c Copyright (C) 1998, 1999 Free Software Foundation, Inc.
@c See the file elisp.texi for copying conditions.
@setfilename ../info/characters
@node Non-ASCII Characters, Searching and Matching, Text, Top
@chapter Non-@sc{ascii} Characters
@cindex multibyte characters
@cindex non-@sc{ascii} characters

  This chapter covers the special issues relating to non-@sc{ascii}
characters and how they are stored in strings and buffers.

@menu
* Text Representations::    Unibyte and multibyte representations
* Converting Representations::  Converting unibyte to multibyte and vice versa.
* Selecting a Representation::  Treating a byte sequence as unibyte or multi.
* Character Codes::         How unibyte and multibyte relate to
                                codes of individual characters.
* Character Sets::          The space of possible characters codes
                                is divided into various character sets.
* Chars and Bytes::         More information about multibyte encodings.
* Splitting Characters::    Converting a character to its byte sequence.
* Scanning Charsets::       Which character sets are used in a buffer?
* Translation of Characters::   Translation tables are used for conversion.
* Coding Systems::          Coding systems are conversions for saving files.
* Input Methods::           Input methods allow users to enter various
                                non-ASCII characters without special keyboards.
* Locales::                 Interacting with the POSIX locale.
@end menu

@node Text Representations
@section Text Representations
@cindex text representations

  Emacs has two @dfn{text representations}---two ways to represent text
in a string or buffer.  These are called @dfn{unibyte} and
@dfn{multibyte}.  Each string, and each buffer, uses one of these two
representations.  For most purposes, you can ignore the issue of
representations, because Emacs converts text between them as
appropriate.  Occasionally in Lisp programming you will need to pay
attention to the difference.

@cindex unibyte text
  In unibyte representation, each character occupies one byte and
therefore the possible character codes range from 0 to 255.  Codes 0
through 127 are @sc{ascii} characters; the codes from 128 through 255
are used for one non-@sc{ascii} character set (you can choose which
character set by setting the variable @code{nonascii-insert-offset}).

@cindex leading code
@cindex multibyte text
@cindex trailing codes
  In multibyte representation, a character may occupy more than one
byte, and as a result, the full range of Emacs character codes can be
stored.  The first byte of a multibyte character is always in the range
128 through 159 (octal 0200 through 0237).  These values are called
@dfn{leading codes}.  The second and subsequent bytes of a multibyte
character are always in the range 160 through 255 (octal 0240 through
0377); these values are @dfn{trailing codes}.

  Some sequences of bytes are not valid in multibyte text: for example,
a single isolated byte in the range 128 through 159 is not allowed.  But
character codes 128 through 159 can appear in multibyte text,
represented as two-byte sequences.  All the character codes 128 through
255 are possible (though slightly abnormal) in multibyte text; they
appear in multibyte buffers and strings when you do explicit encoding
and decoding (@pxref{Explicit Encoding}).

  In a buffer, the buffer-local value of the variable
@code{enable-multibyte-characters} specifies the representation used.
The representation for a string is determined and recorded in the string
when the string is constructed.

@defvar enable-multibyte-characters
This variable specifies the current buffer's text representation.
If it is non-@code{nil}, the buffer contains multibyte text; otherwise,
it contains unibyte text.

You cannot set this variable directly; instead, use the function
@code{set-buffer-multibyte} to change a buffer's representation.
@end defvar

@defvar default-enable-multibyte-characters
This variable's value is entirely equivalent to @code{(default-value
'enable-multibyte-characters)}, and setting this variable changes that
default value.  Setting the local binding of
@code{enable-multibyte-characters} in a specific buffer is not allowed,
but changing the default value is supported, and it is a reasonable
thing to do, because it has no effect on existing buffers.

The @samp{--unibyte} command line option does its job by setting the
default value to @code{nil} early in startup.
@end defvar

@defun position-bytes position
@tindex position-bytes
Return the byte-position corresponding to buffer position @var{position}
in the current buffer.
@end defun

@defun byte-to-position byte-position
@tindex byte-to-position
Return the buffer position corresponding to byte-position
@var{byte-position} in the current buffer.
@end defun

@defun multibyte-string-p string
Return @code{t} if @var{string} is a multibyte string.
@end defun

@node Converting Representations
@section Converting Text Representations

  Emacs can convert unibyte text to multibyte; it can also convert
multibyte text to unibyte, though this conversion loses information.  In
general these conversions happen when inserting text into a buffer, or
when putting text from several strings together in one string.  You can
also explicitly convert a string's contents to either representation.

  Emacs chooses the representation for a string based on the text that
it is constructed from.  The general rule is to convert unibyte text to
multibyte text when combining it with other multibyte text, because the
multibyte representation is more general and can hold whatever
characters the unibyte text has.

  When inserting text into a buffer, Emacs converts the text to the
buffer's representation, as specified by
@code{enable-multibyte-characters} in that buffer.  In particular, when
you insert multibyte text into a unibyte buffer, Emacs converts the text
to unibyte, even though this conversion cannot in general preserve all
the characters that might be in the multibyte text.  The other natural
alternative, to convert the buffer contents to multibyte, is not
acceptable because the buffer's representation is a choice made by the
user that cannot be overridden automatically.

  Converting unibyte text to multibyte text leaves @sc{ascii} characters
unchanged, and likewise character codes 128 through 159.  It converts
the non-@sc{ascii} codes 160 through 255 by adding the value
@code{nonascii-insert-offset} to each character code.  By setting this
variable, you specify which character set the unibyte characters
correspond to (@pxref{Character Sets}).  For example, if
@code{nonascii-insert-offset} is 2048, which is @code{(- (make-char
'latin-iso8859-1) 128)}, then the unibyte non-@sc{ascii} characters
correspond to Latin 1.  If it is 2688, which is @code{(- (make-char
'greek-iso8859-7) 128)}, then they correspond to Greek letters.

  Converting multibyte text to unibyte is simpler: it discards all but
the low 8 bits of each character code.  If @code{nonascii-insert-offset}
has a reasonable value, corresponding to the beginning of some character
set, this conversion is the inverse of the other: converting unibyte
text to multibyte and back to unibyte reproduces the original unibyte
text.

@defvar nonascii-insert-offset
This variable specifies the amount to add to a non-@sc{ascii} character
when converting unibyte text to multibyte.  It also applies when
@code{self-insert-command} inserts a character in the unibyte
non-@sc{ascii} range, 128 through 255.  However, the functions
@code{insert} and @code{insert-char} do not perform this conversion.

The right value to use to select character set @var{cs} is @code{(-
(make-char @var{cs}) 128)}.  If the value of
@code{nonascii-insert-offset} is zero, then conversion actually uses the
value for the Latin 1 character set, rather than zero.
@end defvar

@defvar nonascii-translation-table
This variable provides a more general alternative to
@code{nonascii-insert-offset}.  You can use it to specify independently
how to translate each code in the range of 128 through 255 into a
multibyte character.  The value should be a char-table, or @code{nil}.
If this is non-@code{nil}, it overrides @code{nonascii-insert-offset}.
@end defvar

@defun string-make-unibyte string
This function converts the text of @var{string} to unibyte
representation, if it isn't already, and returns the result.  If
@var{string} is a unibyte string, it is returned unchanged.  Multibyte
character codes are converted to unibyte according to
@code{nonascii-translation-table} or, if that is @code{nil}, using
@code{nonascii-insert-offset}.  If the lookup in the translation table
fails, this function takes just the low 8 bits of each character.
@end defun

@defun string-make-multibyte string
This function converts the text of @var{string} to multibyte
representation, if it isn't already, and returns the result.  If
@var{string} is a multibyte string, it is returned unchanged.
The function @code{unibyte-char-to-multibyte} is used to convert
each unibyte character to a multibyte character.
@end defun

@defun string-to-multibyte string
This function returns a multibyte string containing the same sequence
of character codes as @var{string}.  If @var{string} is a multibyte
string, the value is the equal to @var{string}.
@end defun

@node Selecting a Representation
@section Selecting a Representation

  Sometimes it is useful to examine an existing buffer or string as
multibyte when it was unibyte, or vice versa.

@defun set-buffer-multibyte multibyte
Set the representation type of the current buffer.  If @var{multibyte}
is non-@code{nil}, the buffer becomes multibyte.  If @var{multibyte}
is @code{nil}, the buffer becomes unibyte.

This function leaves the buffer contents unchanged when viewed as a
sequence of bytes.  As a consequence, it can change the contents viewed
as characters; a sequence of two bytes which is treated as one character
in multibyte representation will count as two characters in unibyte
representation.  Character codes 128 through 159 are an exception.  They
are represented by one byte in a unibyte buffer, but when the buffer is
set to multibyte, they are converted to two-byte sequences, and vice
versa.

This function sets @code{enable-multibyte-characters} to record which
representation is in use.  It also adjusts various data in the buffer
(including overlays, text properties and markers) so that they cover the
same text as they did before.

You cannot use @code{set-buffer-multibyte} on an indirect buffer,
because indirect buffers always inherit the representation of the
base buffer.
@end defun

@defun string-as-unibyte string
This function returns a string with the same bytes as @var{string} but
treating each byte as a character.  This means that the value may have
more characters than @var{string} has.

If @var{string} is already a unibyte string, then the value is
@var{string} itself.  Otherwise it is a newly created string, with no
text properties.  If @var{string} is multibyte, any characters it
contains of charset @code{eight-bit-control} or @code{eight-bit-graphic}
are converted to the corresponding single byte.
@end defun

@defun string-as-multibyte string
This function returns a string with the same bytes as @var{string} but
treating each multibyte sequence as one character.  This means that the
value may have fewer characters than @var{string} has.

If @var{string} is already a multibyte string, then the value is
@var{string} itself.  Otherwise it is a newly created string, with no
text properties.  If @var{string} is unibyte and contains any individual
8-bit bytes (i.e.@: not part of a multibyte form), they are converted to
the corresponding multibyte character of charset @code{eight-bit-control}
or @code{eight-bit-graphic}.
@end defun

@node Character Codes
@section Character Codes
@cindex character codes

  The unibyte and multibyte text representations use different character
codes.  The valid character codes for unibyte representation range from
0 to 255---the values that can fit in one byte.  The valid character
codes for multibyte representation range from 0 to 524287, but not all
values in that range are valid.  The values 128 through 255 are not
entirely proper in multibyte text, but they can occur if you do explicit
encoding and decoding (@pxref{Explicit Encoding}).  Some other character
codes cannot occur at all in multibyte text.  Only the @sc{ascii} codes
0 through 127 are completely legitimate in both representations.

@defun char-valid-p charcode &optional genericp
This returns @code{t} if @var{charcode} is valid for either one of the two
text representations.

@example
(char-valid-p 65)
     @result{} t
(char-valid-p 256)
     @result{} nil
(char-valid-p 2248)
     @result{} t
@end example

If the optional argument @var{genericp} is non-@code{nil}, this
function returns @code{t} if @var{charcode} is a generic character
(@pxref{Splitting Characters}).
@end defun

@node Character Sets
@section Character Sets
@cindex character sets

  Emacs classifies characters into various @dfn{character sets}, each of
which has a name which is a symbol.  Each character belongs to one and
only one character set.

  In general, there is one character set for each distinct script.  For
example, @code{latin-iso8859-1} is one character set,
@code{greek-iso8859-7} is another, and @code{ascii} is another.  An
Emacs character set can hold at most 9025 characters; therefore, in some
cases, characters that would logically be grouped together are split
into several character sets.  For example, one set of Chinese
characters, generally known as Big 5, is divided into two Emacs
character sets, @code{chinese-big5-1} and @code{chinese-big5-2}.

  @sc{ascii} characters are in character set @code{ascii}.  The
non-@sc{ascii} characters 128 through 159 are in character set
@code{eight-bit-control}, and codes 160 through 255 are in character set
@code{eight-bit-graphic}.

@defun charsetp object
Returns @code{t} if @var{object} is a symbol that names a character set,
@code{nil} otherwise.
@end defun

@defun charset-list
This function returns a list of all defined character set names.
@end defun

@defun char-charset character
This function returns the name of the character set that @var{character}
belongs to.
@end defun

@defun charset-plist charset
@tindex charset-plist
This function returns the charset property list of the character set
@var{charset}.  Although @var{charset} is a symbol, this is not the same
as the property list of that symbol.  Charset properties are used for
special purposes within Emacs; for example,
@code{preferred-coding-system} helps determine which coding system to
use to encode characters in a charset.
@end defun

@node Chars and Bytes
@section Characters and Bytes
@cindex bytes and characters

@cindex introduction sequence
@cindex dimension (of character set)
  In multibyte representation, each character occupies one or more
bytes.  Each character set has an @dfn{introduction sequence}, which is
normally one or two bytes long.  (Exception: the @sc{ascii} character
set and the @sc{eight-bit-graphic} character set have a zero-length
introduction sequence.)  The introduction sequence is the beginning of
the byte sequence for any character in the character set.  The rest of
the character's bytes distinguish it from the other characters in the
same character set.  Depending on the character set, there are either
one or two distinguishing bytes; the number of such bytes is called the
@dfn{dimension} of the character set.

@defun charset-dimension charset
This function returns the dimension of @var{charset}; at present, the
dimension is always 1 or 2.
@end defun

@defun charset-bytes charset
@tindex charset-bytes
This function returns the number of bytes used to represent a character
in character set @var{charset}.
@end defun

  This is the simplest way to determine the byte length of a character
set's introduction sequence:

@example
(- (charset-bytes @var{charset})
   (charset-dimension @var{charset}))
@end example

@node Splitting Characters
@section Splitting Characters

  The functions in this section convert between characters and the byte
values used to represent them.  For most purposes, there is no need to
be concerned with the sequence of bytes used to represent a character,
because Emacs translates automatically when necessary.

@defun split-char character
Return a list containing the name of the character set of
@var{character}, followed by one or two byte values (integers) which
identify @var{character} within that character set.  The number of byte
values is the character set's dimension.

@example
(split-char 2248)
     @result{} (latin-iso8859-1 72)
(split-char 65)
     @result{} (ascii 65)
(split-char 128)
     @result{} (eight-bit-control 128)
@end example
@end defun

@defun make-char charset &optional code1 code2
This function returns the character in character set @var{charset} whose
position codes are @var{code1} and @var{code2}.  This is roughly the
inverse of @code{split-char}.  Normally, you should specify either one
or both of @var{code1} and @var{code2} according to the dimension of
@var{charset}.  For example,

@example
(make-char 'latin-iso8859-1 72)
     @result{} 2248
@end example
@end defun

@cindex generic characters
  If you call @code{make-char} with no @var{byte-values}, the result is
a @dfn{generic character} which stands for @var{charset}.  A generic
character is an integer, but it is @emph{not} valid for insertion in the
buffer as a character.  It can be used in @code{char-table-range} to
refer to the whole character set (@pxref{Char-Tables}).
@code{char-valid-p} returns @code{nil} for generic characters.
For example:

@example
(make-char 'latin-iso8859-1)
     @result{} 2176
(char-valid-p 2176)
     @result{} nil
(char-valid-p 2176 t)
     @result{} t
(split-char 2176)
     @result{} (latin-iso8859-1 0)
@end example

The character sets @sc{ascii}, @sc{eight-bit-control}, and
@sc{eight-bit-graphic} don't have corresponding generic characters.  If
@var{charset} is one of them and you don't supply @var{code1},
@code{make-char} returns the character code corresponding to the
smallest code in @var{charset}.

@node Scanning Charsets
@section Scanning for Character Sets

  Sometimes it is useful to find out which character sets appear in a
part of a buffer or a string.  One use for this is in determining which
coding systems (@pxref{Coding Systems}) are capable of representing all
of the text in question.

@defun find-charset-region beg end &optional translation
This function returns a list of the character sets that appear in the
current buffer between positions @var{beg} and @var{end}.

The optional argument @var{translation} specifies a translation table to
be used in scanning the text (@pxref{Translation of Characters}).  If it
is non-@code{nil}, then each character in the region is translated
through this table, and the value returned describes the translated
characters instead of the characters actually in the buffer.
@end defun

@defun find-charset-string string &optional translation
This function returns a list of the character sets that appear in the
string @var{string}.  It is just like @code{find-charset-region}, except
that it applies to the contents of @var{string} instead of part of the
current buffer.
@end defun

@node Translation of Characters
@section Translation of Characters
@cindex character translation tables
@cindex translation tables

  A @dfn{translation table} specifies a mapping of characters
into characters.  These tables are used in encoding and decoding, and
for other purposes.  Some coding systems specify their own particular
translation tables; there are also default translation tables which
apply to all other coding systems.

@defun make-translation-table &rest translations
This function returns a translation table based on the argument
@var{translations}.  Each element of @var{translations} should be a
list of elements of the form @code{(@var{from} . @var{to})}; this says
to translate the character @var{from} into @var{to}.

The arguments and the forms in each argument are processed in order,
and if a previous form already translates @var{to} to some other
character, say @var{to-alt}, @var{from} is also translated to
@var{to-alt}.

You can also map one whole character set into another character set with
the same dimension.  To do this, you specify a generic character (which
designates a character set) for @var{from} (@pxref{Splitting Characters}).
In this case, @var{to} should also be a generic character, for another
character set of the same dimension.  Then the translation table
translates each character of @var{from}'s character set into the
corresponding character of @var{to}'s character set.
@end defun

  In decoding, the translation table's translations are applied to the
characters that result from ordinary decoding.  If a coding system has
property @code{character-translation-table-for-decode}, that specifies
the translation table to use.  Otherwise, if
@code{standard-translation-table-for-decode} is non-@code{nil}, decoding
uses that table.

  In encoding, the translation table's translations are applied to the
characters in the buffer, and the result of translation is actually
encoded.  If a coding system has property
@code{character-translation-table-for-encode}, that specifies the
translation table to use.  Otherwise the variable
@code{standard-translation-table-for-encode} specifies the translation
table.

@defvar standard-translation-table-for-decode
This is the default translation table for decoding, for
coding systems that don't specify any other translation table.
@end defvar

@defvar standard-translation-table-for-encode
This is the default translation table for encoding, for
coding systems that don't specify any other translation table.
@end defvar

@defvar translation-table-for-input
Self-inserting characters are translated through this translation
table before they are inserted.
@end defvar

@node Coding Systems
@section Coding Systems

@cindex coding system
  When Emacs reads or writes a file, and when Emacs sends text to a
subprocess or receives text from a subprocess, it normally performs
character code conversion and end-of-line conversion as specified
by a particular @dfn{coding system}.

  How to define a coding system is an arcane matter, and is not
documented here.

@menu
* Coding System Basics::        Basic concepts.
* Encoding and I/O::            How file I/O functions handle coding systems.
* Lisp and Coding Systems::     Functions to operate on coding system names.
* User-Chosen Coding Systems::  Asking the user to choose a coding system.
* Default Coding Systems::      Controlling the default choices.
* Specifying Coding Systems::   Requesting a particular coding system
                                    for a single file operation.
* Explicit Encoding::           Encoding or decoding text without doing I/O.
* Terminal I/O Encoding::       Use of encoding for terminal I/O.
* MS-DOS File Types::           How DOS "text" and "binary" files
                                    relate to coding systems.
@end menu

@node Coding System Basics
@subsection Basic Concepts of Coding Systems

@cindex character code conversion
  @dfn{Character code conversion} involves conversion between the encoding
used inside Emacs and some other encoding.  Emacs supports many
different encodings, in that it can convert to and from them.  For
example, it can convert text to or from encodings such as Latin 1, Latin
2, Latin 3, Latin 4, Latin 5, and several variants of ISO 2022.  In some
cases, Emacs supports several alternative encodings for the same
characters; for example, there are three coding systems for the Cyrillic
(Russian) alphabet: ISO, Alternativnyj, and KOI8.

  Most coding systems specify a particular character code for
conversion, but some of them leave the choice unspecified---to be chosen
heuristically for each file, based on the data.

@cindex end of line conversion
  @dfn{End of line conversion} handles three different conventions used
on various systems for representing end of line in files.  The Unix
convention is to use the linefeed character (also called newline).  The
DOS convention is to use a carriage-return and a linefeed at the end of
a line.  The Mac convention is to use just carriage-return.

@cindex base coding system
@cindex variant coding system
  @dfn{Base coding systems} such as @code{latin-1} leave the end-of-line
conversion unspecified, to be chosen based on the data.  @dfn{Variant
coding systems} such as @code{latin-1-unix}, @code{latin-1-dos} and
@code{latin-1-mac} specify the end-of-line conversion explicitly as
well.  Most base coding systems have three corresponding variants whose
names are formed by adding @samp{-unix}, @samp{-dos} and @samp{-mac}.

  The coding system @code{raw-text} is special in that it prevents
character code conversion, and causes the buffer visited with that
coding system to be a unibyte buffer.  It does not specify the
end-of-line conversion, allowing that to be determined as usual by the
data, and has the usual three variants which specify the end-of-line
conversion.  @code{no-conversion} is equivalent to @code{raw-text-unix}:
it specifies no conversion of either character codes or end-of-line.

  The coding system @code{emacs-mule} specifies that the data is
represented in the internal Emacs encoding.  This is like
@code{raw-text} in that no code conversion happens, but different in
that the result is multibyte data.

@defun coding-system-get coding-system property
This function returns the specified property of the coding system
@var{coding-system}.  Most coding system properties exist for internal
purposes, but one that you might find useful is @code{mime-charset}.
That property's value is the name used in MIME for the character coding
which this coding system can read and write.  Examples:

@example
(coding-system-get 'iso-latin-1 'mime-charset)
     @result{} iso-8859-1
(coding-system-get 'iso-2022-cn 'mime-charset)
     @result{} iso-2022-cn
(coding-system-get 'cyrillic-koi8 'mime-charset)
     @result{} koi8-r
@end example

The value of the @code{mime-charset} property is also defined
as an alias for the coding system.
@end defun

@node Encoding and I/O
@subsection Encoding and I/O

  The principal purpose of coding systems is for use in reading and
writing files.  The function @code{insert-file-contents} uses
a coding system for decoding the file data, and @code{write-region}
uses one to encode the buffer contents.

  You can specify the coding system to use either explicitly
(@pxref{Specifying Coding Systems}), or implicitly using the defaulting
mechanism (@pxref{Default Coding Systems}).  But these methods may not
completely specify what to do.  For example, they may choose a coding
system such as @code{undefined} which leaves the character code
conversion to be determined from the data.  In these cases, the I/O
operation finishes the job of choosing a coding system.  Very often
you will want to find out afterwards which coding system was chosen.

@defvar buffer-file-coding-system
This variable records the coding system that was used for visiting the
current buffer.  It is used for saving the buffer, and for writing part
of the buffer with @code{write-region}.  If the text to be written
cannot be safely encoded using the coding system specified by this
variable, these operations select an alternative encoding by calling
the function @code{select-safe-coding-system} (@pxref{User-Chosen
Coding Systems}).  If selecting a different encoding requires to ask
the user to specify a coding system, @code{buffer-file-coding-system}
is updated to the newly selected coding system.

@code{buffer-file-coding-system} does @emph{not} affect sending text
to a subprocess.
@end defvar

@defvar save-buffer-coding-system
This variable specifies the coding system for saving the buffer (by
overriding @code{buffer-file-coding-system}).  Note that it is not used
for @code{write-region}.

When a command to save the buffer starts out to use
@code{buffer-file-coding-system} (or @code{save-buffer-coding-system}),
and that coding system cannot handle
the actual text in the buffer, the command asks the user to choose
another coding system (by calling @code{select-safe-coding-system}).
After that happens, the command also updates
@code{buffer-file-coding-system} to represent the coding system that
the user specified.
@end defvar

@defvar last-coding-system-used
I/O operations for files and subprocesses set this variable to the
coding system name that was used.  The explicit encoding and decoding
functions (@pxref{Explicit Encoding}) set it too.

@strong{Warning:} Since receiving subprocess output sets this variable,
it can change whenever Emacs waits; therefore, you should copy the
value shortly after the function call that stores the value you are
interested in.
@end defvar

  The variable @code{selection-coding-system} specifies how to encode
selections for the window system.  @xref{Window System Selections}.

@node Lisp and Coding Systems
@subsection Coding Systems in Lisp

  Here are the Lisp facilities for working with coding systems:

@defun coding-system-list &optional base-only
This function returns a list of all coding system names (symbols).  If
@var{base-only} is non-@code{nil}, the value includes only the
base coding systems.  Otherwise, it includes alias and variant coding
systems as well.
@end defun

@defun coding-system-p object
This function returns @code{t} if @var{object} is a coding system
name.
@end defun

@defun check-coding-system coding-system
This function checks the validity of @var{coding-system}.
If that is valid, it returns @var{coding-system}.
Otherwise it signals an error with condition @code{coding-system-error}.
@end defun

@defun coding-system-change-eol-conversion coding-system eol-type
This function returns a coding system which is like @var{coding-system}
except for its eol conversion, which is specified by @code{eol-type}.
@var{eol-type} should be @code{unix}, @code{dos}, @code{mac}, or
@code{nil}.  If it is @code{nil}, the returned coding system determines
the end-of-line conversion from the data.
@end defun

@defun coding-system-change-text-conversion eol-coding text-coding
This function returns a coding system which uses the end-of-line
conversion of @var{eol-coding}, and the text conversion of
@var{text-coding}.  If @var{text-coding} is @code{nil}, it returns
@code{undecided}, or one of its variants according to @var{eol-coding}.
@end defun

@defun find-coding-systems-region from to
This function returns a list of coding systems that could be used to
encode a text between @var{from} and @var{to}.  All coding systems in
the list can safely encode any multibyte characters in that portion of
the text.

If the text contains no multibyte characters, the function returns the
list @code{(undecided)}.
@end defun

@defun find-coding-systems-string string
This function returns a list of coding systems that could be used to
encode the text of @var{string}.  All coding systems in the list can
safely encode any multibyte characters in @var{string}.  If the text
contains no multibyte characters, this returns the list
@code{(undecided)}.
@end defun

@defun find-coding-systems-for-charsets charsets
This function returns a list of coding systems that could be used to
encode all the character sets in the list @var{charsets}.
@end defun

@defun detect-coding-region start end &optional highest
This function chooses a plausible coding system for decoding the text
from @var{start} to @var{end}.  This text should be a byte sequence
(@pxref{Explicit Encoding}).

Normally this function returns a list of coding systems that could
handle decoding the text that was scanned.  They are listed in order of
decreasing priority.  But if @var{highest} is non-@code{nil}, then the
return value is just one coding system, the one that is highest in
priority.

If the region contains only @sc{ascii} characters, the value
is @code{undecided} or @code{(undecided)}.
@end defun

@defun detect-coding-string string highest
This function is like @code{detect-coding-region} except that it
operates on the contents of @var{string} instead of bytes in the buffer.
@end defun

  @xref{Process Information}, for how to examine or set the coding
systems used for I/O to a subprocess.

@node User-Chosen Coding Systems
@subsection User-Chosen Coding Systems

@cindex select safe coding system
@defun select-safe-coding-system from to &optional default-coding-system accept-default-p
This function selects a coding system for encoding specified text,
asking the user to choose if necessary.  Normally the specified text
is the text in the current buffer between @var{from} and @var{to},
defaulting to the whole buffer if they are @code{nil}.  If @var{from}
is a string, the string specifies the text to encode, and @var{to} is
ignored.

If @var{default-coding-system} is non-@code{nil}, that is the first
coding system to try; if that can handle the text,
@code{select-safe-coding-system} returns that coding system.  It can
also be a list of coding systems; then the function tries each of them
one by one.  After trying all of them, it next tries the user's most
preferred coding system (@pxref{Recognize Coding,
prefer-coding-system, the description of @code{prefer-coding-system},
emacs, GNU Emacs Manual}), and after that the current buffer's value
of @code{buffer-file-coding-system} (if it is not @code{undecided}).

If one of those coding systems can safely encode all the specified
text, @code{select-safe-coding-system} chooses it and returns it.
Otherwise, it asks the user to choose from a list of coding systems
which can encode all the text, and returns the user's choice.

The optional argument @var{accept-default-p}, if non-@code{nil},
should be a function to determine whether the coding system selected
without user interaction is acceptable.  If this function returns
@code{nil}, the silently selected coding system is rejected, and the
user is asked to select a coding system from a list of possible
candidates.

@vindex select-safe-coding-system-accept-default-p
If the variable @code{select-safe-coding-system-accept-default-p} is
non-@code{nil}, its value overrides the value of
@var{accept-default-p}.
@end defun

  Here are two functions you can use to let the user specify a coding
system, with completion.  @xref{Completion}.

@defun read-coding-system prompt &optional default
This function reads a coding system using the minibuffer, prompting with
string @var{prompt}, and returns the coding system name as a symbol.  If
the user enters null input, @var{default} specifies which coding system
to return.  It should be a symbol or a string.
@end defun

@defun read-non-nil-coding-system prompt
This function reads a coding system using the minibuffer, prompting with
string @var{prompt}, and returns the coding system name as a symbol.  If
the user tries to enter null input, it asks the user to try again.
@xref{Coding Systems}.
@end defun

@node Default Coding Systems
@subsection Default Coding Systems

  This section describes variables that specify the default coding
system for certain files or when running certain subprograms, and the
function that I/O operations use to access them.

  The idea of these variables is that you set them once and for all to the
defaults you want, and then do not change them again.  To specify a
particular coding system for a particular operation in a Lisp program,
don't change these variables; instead, override them using
@code{coding-system-for-read} and @code{coding-system-for-write}
(@pxref{Specifying Coding Systems}).

@defvar auto-coding-regexp-alist
This variable is an alist of text patterns and corresponding coding
systems. Each element has the form @code{(@var{regexp}
. @var{coding-system})}; a file whose first few kilobytes match
@var{regexp} is decoded with @var{coding-system} when its contents are
read into a buffer.  The settings in this alist take priority over
@code{coding:} tags in the files and the contents of
@code{file-coding-system-alist} (see below).  The default value is set
so that Emacs automatically recognizes mail files in Babyl format and
reads them with no code conversions.
@end defvar

@defvar file-coding-system-alist
This variable is an alist that specifies the coding systems to use for
reading and writing particular files.  Each element has the form
@code{(@var{pattern} . @var{coding})}, where @var{pattern} is a regular
expression that matches certain file names.  The element applies to file
names that match @var{pattern}.

The @sc{cdr} of the element, @var{coding}, should be either a coding
system, a cons cell containing two coding systems, or a function name (a
symbol with a function definition).  If @var{coding} is a coding system,
that coding system is used for both reading the file and writing it.  If
@var{coding} is a cons cell containing two coding systems, its @sc{car}
specifies the coding system for decoding, and its @sc{cdr} specifies the
coding system for encoding.

If @var{coding} is a function name, the function must return a coding
system or a cons cell containing two coding systems.  This value is used
as described above.
@end defvar

@defvar process-coding-system-alist
This variable is an alist specifying which coding systems to use for a
subprocess, depending on which program is running in the subprocess.  It
works like @code{file-coding-system-alist}, except that @var{pattern} is
matched against the program name used to start the subprocess.  The coding
system or systems specified in this alist are used to initialize the
coding systems used for I/O to the subprocess, but you can specify
other coding systems later using @code{set-process-coding-system}.
@end defvar

  @strong{Warning:} Coding systems such as @code{undecided}, which
determine the coding system from the data, do not work entirely reliably
with asynchronous subprocess output.  This is because Emacs handles
asynchronous subprocess output in batches, as it arrives.  If the coding
system leaves the character code conversion unspecified, or leaves the
end-of-line conversion unspecified, Emacs must try to detect the proper
conversion from one batch at a time, and this does not always work.

  Therefore, with an asynchronous subprocess, if at all possible, use a
coding system which determines both the character code conversion and
the end of line conversion---that is, one like @code{latin-1-unix},
rather than @code{undecided} or @code{latin-1}.

@defvar network-coding-system-alist
This variable is an alist that specifies the coding system to use for
network streams.  It works much like @code{file-coding-system-alist},
with the difference that the @var{pattern} in an element may be either a
port number or a regular expression.  If it is a regular expression, it
is matched against the network service name used to open the network
stream.
@end defvar

@defvar default-process-coding-system
This variable specifies the coding systems to use for subprocess (and
network stream) input and output, when nothing else specifies what to
do.

The value should be a cons cell of the form @code{(@var{input-coding}
. @var{output-coding})}.  Here @var{input-coding} applies to input from
the subprocess, and @var{output-coding} applies to output to it.
@end defvar

@defvar auto-coding-functions
This variable holds a list of functions that try to determine a
coding system for a file based on its undecoded contents.

Each function in this list should be written to look at text in the
current buffer, but should not modify it in any way.  The buffer will
contain undecoded text of parts of the file.  Each function should
take one argument, @var{size}, which tells it how many characters to
look at, starting from point.  If the function succeeds in determining
a coding system for the file, it should return that coding system.
Otherwise, it should return @code{nil}.

If a file has a @samp{coding:} tag, that takes precedence, so these
functions won't be called.
@end defvar

@defun find-operation-coding-system operation &rest arguments
This function returns the coding system to use (by default) for
performing @var{operation} with @var{arguments}.  The value has this
form:

@example
(@var{decoding-system} @var{encoding-system})
@end example

The first element, @var{decoding-system}, is the coding system to use
for decoding (in case @var{operation} does decoding), and
@var{encoding-system} is the coding system for encoding (in case
@var{operation} does encoding).

The argument @var{operation} should be a symbol, one of
@code{insert-file-contents}, @code{write-region}, @code{call-process},
@code{call-process-region}, @code{start-process}, or
@code{open-network-stream}.  These are the names of the Emacs I/O primitives
that can do coding system conversion.

The remaining arguments should be the same arguments that might be given
to that I/O primitive.  Depending on the primitive, one of those
arguments is selected as the @dfn{target}.  For example, if
@var{operation} does file I/O, whichever argument specifies the file
name is the target.  For subprocess primitives, the process name is the
target.  For @code{open-network-stream}, the target is the service name
or port number.

This function looks up the target in @code{file-coding-system-alist},
@code{process-coding-system-alist}, or
@code{network-coding-system-alist}, depending on @var{operation}.
@xref{Default Coding Systems}.
@end defun

@node Specifying Coding Systems
@subsection Specifying a Coding System for One Operation

  You can specify the coding system for a specific operation by binding
the variables @code{coding-system-for-read} and/or
@code{coding-system-for-write}.

@defvar coding-system-for-read
If this variable is non-@code{nil}, it specifies the coding system to
use for reading a file, or for input from a synchronous subprocess.

It also applies to any asynchronous subprocess or network stream, but in
a different way: the value of @code{coding-system-for-read} when you
start the subprocess or open the network stream specifies the input
decoding method for that subprocess or network stream.  It remains in
use for that subprocess or network stream unless and until overridden.

The right way to use this variable is to bind it with @code{let} for a
specific I/O operation.  Its global value is normally @code{nil}, and
you should not globally set it to any other value.  Here is an example
of the right way to use the variable:

@example
;; @r{Read the file with no character code conversion.}
;; @r{Assume @sc{crlf} represents end-of-line.}
(let ((coding-system-for-write 'emacs-mule-dos))
  (insert-file-contents filename))
@end example

When its value is non-@code{nil}, @code{coding-system-for-read} takes
precedence over all other methods of specifying a coding system to use for
input, including @code{file-coding-system-alist},
@code{process-coding-system-alist} and
@code{network-coding-system-alist}.
@end defvar

@defvar coding-system-for-write
This works much like @code{coding-system-for-read}, except that it
applies to output rather than input.  It affects writing to files,
as well as sending output to subprocesses and net connections.

When a single operation does both input and output, as do
@code{call-process-region} and @code{start-process}, both
@code{coding-system-for-read} and @code{coding-system-for-write}
affect it.
@end defvar

@defvar inhibit-eol-conversion
When this variable is non-@code{nil}, no end-of-line conversion is done,
no matter which coding system is specified.  This applies to all the
Emacs I/O and subprocess primitives, and to the explicit encoding and
decoding functions (@pxref{Explicit Encoding}).
@end defvar

@node Explicit Encoding
@subsection Explicit Encoding and Decoding
@cindex encoding text
@cindex decoding text

  All the operations that transfer text in and out of Emacs have the
ability to use a coding system to encode or decode the text.
You can also explicitly encode and decode text using the functions
in this section.

  The result of encoding, and the input to decoding, are not ordinary
text.  They logically consist of a series of byte values; that is, a
series of characters whose codes are in the range 0 through 255.  In a
multibyte buffer or string, character codes 128 through 159 are
represented by multibyte sequences, but this is invisible to Lisp
programs.

  The usual way to read a file into a buffer as a sequence of bytes, so
you can decode the contents explicitly, is with
@code{insert-file-contents-literally} (@pxref{Reading from Files});
alternatively, specify a non-@code{nil} @var{rawfile} argument when
visiting a file with @code{find-file-noselect}.  These methods result in
a unibyte buffer.

  The usual way to use the byte sequence that results from explicitly
encoding text is to copy it to a file or process---for example, to write
it with @code{write-region} (@pxref{Writing to Files}), and suppress
encoding by binding @code{coding-system-for-write} to
@code{no-conversion}.

  Here are the functions to perform explicit encoding or decoding.  The
decoding functions produce sequences of bytes; the encoding functions
are meant to operate on sequences of bytes.  All of these functions
discard text properties.

@defun encode-coding-region start end coding-system
This function encodes the text from @var{start} to @var{end} according
to coding system @var{coding-system}.  The encoded text replaces the
original text in the buffer.  The result of encoding is logically a
sequence of bytes, but the buffer remains multibyte if it was multibyte
before.
@end defun

@defun encode-coding-string string coding-system
This function encodes the text in @var{string} according to coding
system @var{coding-system}.  It returns a new string containing the
encoded text.  The result of encoding is a unibyte string.
@end defun

@defun decode-coding-region start end coding-system
This function decodes the text from @var{start} to @var{end} according
to coding system @var{coding-system}.  The decoded text replaces the
original text in the buffer.  To make explicit decoding useful, the text
before decoding ought to be a sequence of byte values, but both
multibyte and unibyte buffers are acceptable.
@end defun

@defun decode-coding-string string coding-system
This function decodes the text in @var{string} according to coding
system @var{coding-system}.  It returns a new string containing the
decoded text.  To make explicit decoding useful, the contents of
@var{string} ought to be a sequence of byte values, but a multibyte
string is acceptable.
@end defun

@defun decode-coding-inserted-region from to filename &optional visit beg end replace
This function decodes the text from @var{from} to @var{to} as if
it were being read from file @var{filename} using @code{insert-file-contents}
using the rest of the arguments provided.

The normal way to use this function is after reading text from a file
without decoding, if you decide you would rather have decoded it.
Instead of deleting the text and reading it again, this time with
decoding, you can call this function.
@end defun

@node Terminal I/O Encoding
@subsection Terminal I/O Encoding

  Emacs can decode keyboard input using a coding system, and encode
terminal output.  This is useful for terminals that transmit or display
text using a particular encoding such as Latin-1.  Emacs does not set
@code{last-coding-system-used} for encoding or decoding for the
terminal.

@defun keyboard-coding-system
This function returns the coding system that is in use for decoding
keyboard input---or @code{nil} if no coding system is to be used.
@end defun

@defun set-keyboard-coding-system coding-system
This function specifies @var{coding-system} as the coding system to
use for decoding keyboard input.  If @var{coding-system} is @code{nil},
that means do not decode keyboard input.
@end defun

@defun terminal-coding-system
This function returns the coding system that is in use for encoding
terminal output---or @code{nil} for no encoding.
@end defun

@defun set-terminal-coding-system coding-system
This function specifies @var{coding-system} as the coding system to use
for encoding terminal output.  If @var{coding-system} is @code{nil},
that means do not encode terminal output.
@end defun

@node MS-DOS File Types
@subsection MS-DOS File Types
@cindex DOS file types
@cindex MS-DOS file types
@cindex Windows file types
@cindex file types on MS-DOS and Windows
@cindex text files and binary files
@cindex binary files and text files

  On MS-DOS and Microsoft Windows, Emacs guesses the appropriate
end-of-line conversion for a file by looking at the file's name.  This
feature classifies files as @dfn{text files} and @dfn{binary files}.  By
``binary file'' we mean a file of literal byte values that are not
necessarily meant to be characters; Emacs does no end-of-line conversion
and no character code conversion for them.  On the other hand, the bytes
in a text file are intended to represent characters; when you create a
new file whose name implies that it is a text file, Emacs uses DOS
end-of-line conversion.

@defvar buffer-file-type
This variable, automatically buffer-local in each buffer, records the
file type of the buffer's visited file.  When a buffer does not specify
a coding system with @code{buffer-file-coding-system}, this variable is
used to determine which coding system to use when writing the contents
of the buffer.  It should be @code{nil} for text, @code{t} for binary.
If it is @code{t}, the coding system is @code{no-conversion}.
Otherwise, @code{undecided-dos} is used.

Normally this variable is set by visiting a file; it is set to
@code{nil} if the file was visited without any actual conversion.
@end defvar

@defopt file-name-buffer-file-type-alist
This variable holds an alist for recognizing text and binary files.
Each element has the form (@var{regexp} . @var{type}), where
@var{regexp} is matched against the file name, and @var{type} may be
@code{nil} for text, @code{t} for binary, or a function to call to
compute which.  If it is a function, then it is called with a single
argument (the file name) and should return @code{t} or @code{nil}.

When running on MS-DOS or MS-Windows, Emacs checks this alist to decide
which coding system to use when reading a file.  For a text file,
@code{undecided-dos} is used.  For a binary file, @code{no-conversion}
is used.

If no element in this alist matches a given file name, then
@code{default-buffer-file-type} says how to treat the file.
@end defopt

@defopt default-buffer-file-type
This variable says how to handle files for which
@code{file-name-buffer-file-type-alist} says nothing about the type.

If this variable is non-@code{nil}, then these files are treated as
binary: the coding system @code{no-conversion} is used.  Otherwise,
nothing special is done for them---the coding system is deduced solely
from the file contents, in the usual Emacs fashion.
@end defopt

@node Input Methods
@section Input Methods
@cindex input methods

  @dfn{Input methods} provide convenient ways of entering non-@sc{ascii}
characters from the keyboard.  Unlike coding systems, which translate
non-@sc{ascii} characters to and from encodings meant to be read by
programs, input methods provide human-friendly commands.  (@xref{Input
Methods,,, emacs, The GNU Emacs Manual}, for information on how users
use input methods to enter text.)  How to define input methods is not
yet documented in this manual, but here we describe how to use them.

  Each input method has a name, which is currently a string;
in the future, symbols may also be usable as input method names.

@defvar current-input-method
This variable holds the name of the input method now active in the
current buffer.  (It automatically becomes local in each buffer when set
in any fashion.)  It is @code{nil} if no input method is active in the
buffer now.
@end defvar

@defvar default-input-method
This variable holds the default input method for commands that choose an
input method.  Unlike @code{current-input-method}, this variable is
normally global.
@end defvar

@defun set-input-method input-method
This function activates input method @var{input-method} for the current
buffer.  It also sets @code{default-input-method} to @var{input-method}.
If @var{input-method} is @code{nil}, this function deactivates any input
method for the current buffer.
@end defun

@defun read-input-method-name prompt &optional default inhibit-null
This function reads an input method name with the minibuffer, prompting
with @var{prompt}.  If @var{default} is non-@code{nil}, that is returned
by default, if the user enters empty input.  However, if
@var{inhibit-null} is non-@code{nil}, empty input signals an error.

The returned value is a string.
@end defun

@defvar input-method-alist
This variable defines all the supported input methods.
Each element defines one input method, and should have the form:

@example
(@var{input-method} @var{language-env} @var{activate-func}
 @var{title} @var{description} @var{args}...)
@end example

Here @var{input-method} is the input method name, a string;
@var{language-env} is another string, the name of the language
environment this input method is recommended for.  (That serves only for
documentation purposes.)

@var{activate-func} is a function to call to activate this method.  The
@var{args}, if any, are passed as arguments to @var{activate-func}.  All
told, the arguments to @var{activate-func} are @var{input-method} and
the @var{args}.

@var{title} is a string to display in the mode line while this method is
active.  @var{description} is a string describing this method and what
it is good for.
@end defvar

  The fundamental interface to input methods is through the
variable @code{input-method-function}.  @xref{Reading One Event}.

@node Locales
@section Locales
@cindex locale

  POSIX defines a concept of ``locales'' which control which language
to use in language-related features.  These Emacs variables control
how Emacs interacts with these features.

@defvar locale-coding-system
@tindex locale-coding-system
@cindex keyboard input decoding on X
This variable specifies the coding system to use for decoding system
error messages and---on X Window system only---keyboard input, for
encoding the format argument to @code{format-time-string}, and for
decoding the return value of @code{format-time-string}.
@end defvar

@defvar system-messages-locale
@tindex system-messages-locale
This variable specifies the locale to use for generating system error
messages.  Changing the locale can cause messages to come out in a
different language or in a different orthography.  If the variable is
@code{nil}, the locale is specified by environment variables in the
usual POSIX fashion.
@end defvar

@defvar system-time-locale
@tindex system-time-locale
This variable specifies the locale to use for formatting time values.
Changing the locale can cause messages to appear according to the
conventions of a different language.  If the variable is @code{nil}, the
locale is specified by environment variables in the usual POSIX fashion.
@end defvar

@defun locale-info item
This function returns locale data @var{item} for the current POSIX
locale, if available.  @var{item} should be one of these symbols:

@table @code
@item codeset
Return the character set as a string (locale item @code{CODESET}).

@item days
Return a 7-element vector of day names (locale items
@code{DAY_1} through @code{DAY_7});

@item months
Return a 12-element vector of month names (locale items @code{MON_1}
through @code{MON_12}).

@item paper
Return a list @code{(@var{width} @var{height})} for the default paper
size measured in milimeters (locale items @code{PAPER_WIDTH} and
@code{PAPER_HEIGHT}).
@end table

If the system can't provide the requested information, or if
@var{item} is not one of those symbols, the value is @code{nil}.  All
strings in the return value are decoded using
@code{locale-coding-system}.  @xref{Locales,,, libc, GNU Libc Manual},
for more information about locales and locale items.
@end defun