+++ /dev/null
-\input texinfo
-@setfilename mbapi.info
-@settitle Multibyte API
-@setchapternewpage off
-
-@c Open issues:
-
-@c What's the best way to report errors? Should functions return a
-@c magic value, according to C tradition, or should they signal a
-@c Guile exception?
-
-@c
-
-
-@node Working With Multibyte Strings in C
-@chapter Working With Multibyte Strings in C
-
-Guile allows strings to contain characters drawn from a wide variety of
-languages, including many Asian, Eastern European, and Middle Eastern
-languages, in a uniform and unrestricted way. The string representation
-normally used in C code --- an array of @sc{ASCII} characters --- is not
-sufficient for Guile strings, since they may contain characters not
-present in @sc{ASCII}.
-
-Instead, Guile uses a very large character set, and encodes each
-character as a sequence of one or more bytes. We call this
-variable-width encoding a @dfn{multibyte} encoding. Guile uses this
-single encoding internally for all strings, symbol names, error
-messages, etc., and performs appropriate conversions upon input and
-output.
-
-The use of this variable-width encoding is almost invisible to Scheme
-code. Strings are still indexed by character number, not by byte
-offset; @code{string-length} still returns the length of a string in
-characters, not in bytes. @code{string-ref} and @code{string-set!} are
-no longer guaranteed to be constant-time operations, but Guile uses
-various strategies to reduce the impact of this change.
-
-However, the encoding is visible via Guile's C interface, which gives
-the user direct access to a string's bytes. This chapter explains how
-to work with Guile multibyte text in C code. Since variable-width
-encodings are clumsier to work with than simple fixed-width encodings,
-Guile provides a set of standard macros and functions for manipulating
-multibyte text to make the job easier. Furthermore, Guile makes some
-promises about the encoding which you can use in writing your own text
-processing code.
-
-While we discuss guaranteed properties of Guile's encoding, and provide
-functions to operate on its character set, we do not actually specify
-either the character set or encoding here. This is because we expect
-both of them to change in the future: currently, Guile uses the same
-encoding as GNU Emacs 20.4, but we hope to change Guile (and GNU Emacs
-as well) to use Unicode and UTF-8, with some extensions. This will make
-it more comfortable to use Guile with other systems which use UTF-8,
-like the GTk user interface toolkit.
-
-@menu
-* Multibyte String Terminology::
-* Promised Properties of the Guile Multibyte Encoding::
-* Functions for Operating on Multibyte Text::
-* Multibyte Text Processing Errors::
-* Why Guile Does Not Use a Fixed-Width Encoding::
-@end menu
-
-
-@node Multibyte String Terminology, Promised Properties of the Guile Multibyte Encoding, Working With Multibyte Strings in C, Working With Multibyte Strings in C
-@section Multibyte String Terminology
-
-In the descriptions which follow, we make the following definitions:
-@table @dfn
-
-@item byte
-A @dfn{byte} is a number between 0 and 255. It has no inherent textual
-interpretation. So 65 is a byte, not a character.
-
-@item character
-A @dfn{character} is a unit of text. It has no inherent numeric value.
-@samp{A} and @samp{.} are characters, not bytes. (This is different
-from the C language's definition of @dfn{character}; in this chapter, we
-will always use a phrase like ``the C language's @code{char} type'' when
-that's what we mean.)
-
-@item character set
-A @dfn{character set} is an invertible mapping between numbers and a
-given set of characters. @sc{ASCII} is a character set assigning
-characters to the numbers 0 through 127. It maps @samp{A} onto the
-number 65, and @samp{.} onto 46.
-
-Note that a character set maps characters onto numbers, @emph{not
-necessarily} onto bytes. For example, the Unicode character set maps
-the Greek lower-case @samp{alpha} character onto the number 945, which
-is not a byte.
-
-(This is what Internet standards would call a "coding character set".)
-
-@item encoding
-An encoding maps numbers onto sequences of bytes. For example, the
-UTF-8 encoding, defined in the Unicode Standard, would map the number
-945 onto the sequence of bytes @samp{206 177}. When using the
-@sc{ASCII} character set, every number assigned also happens to be a
-byte, so there is an obvious trivial encoding for @sc{ASCII} in bytes.
-
-(This is what Internet standards would call a "character encoding
-scheme".)
-
-@end table
-
-Thus, to turn a character into a sequence of bytes, you need a character
-set to assign a number to that character, and then an encoding to turn
-that number into a sequence of bytes.
-
-Likewise, to interpret a sequence of bytes as a sequence of characters,
-you use an encoding to extract a sequence of numbers from the bytes, and
-then a character set to turn the numbers into characters.
-
-Errors can occur while carrying out either of these processes. For
-example, under a particular encoding, a given string of bytes might not
-correspond to any number. For example, the byte sequence @samp{128 128}
-is not a valid encoding of any number under UTF-8.
-
-Having carefully defined our terminology, we will now abuse it.
-
-We will sometimes use the word @dfn{character} to refer to the number
-assigned to a character by a character set, in contexts where it's
-obvious we mean a number.
-
-Sometimes there is a close association between a particular encoding and
-a particular character set. Thus, we may sometimes refer to the
-character set and encoding together as an @dfn{encoding}.
-
-
-@node Promised Properties of the Guile Multibyte Encoding, Functions for Operating on Multibyte Text, Multibyte String Terminology, Working With Multibyte Strings in C
-@section Promised Properties of the Guile Multibyte Encoding
-
-Internally, Guile uses a single encoding for all text --- symbols,
-strings, error messages, etc. Here we list a number of helpful
-properties of Guile's encoding. It is correct to write code which
-assumes these properties; code which uses these assumptions will be
-portable to all future versions of Guile, as far as we know.
-
-@b{Every @sc{ASCII} character is encoded as a single byte from 0 to 127, in
-the obvious way.} This means that a standard C string containing only
-@sc{ASCII} characters is a valid Guile string (except for the terminator;
-Guile strings store the length explicitly, so they can contain null
-characters).
-
-@b{The encodings of non-@sc{ASCII} characters use only bytes between 128
-and 255.} That is, when we turn a non-@sc{ASCII} character into a
-series of bytes, none of those bytes can ever be mistaken for the
-encoding of an @sc{ASCII} character. This means that you can search a
-Guile string for an @sc{ASCII} character using the standard
-@code{memchr} library function. By extension, you can search for an
-@sc{ASCII} substring in a Guile string using a traditional substring
-search algorithm --- you needn't add special checks to verify encoding
-boundaries, etc.
-
-@b{No character encoding is a subsequence of any other character
-encoding.} (This is just a stronger version of the previous promise.)
-This means that you can search for occurrences of one Guile string
-within another Guile string just as if they were raw byte strings. You
-can use the stock @code{memmem} function (provided on GNU systems, at
-least) for such searches. If you don't need the ability to represent
-null characters in your text, you can still use null-termination for
-strings, and use the traditional string-handling functions like
-@code{strlen}, @code{strstr}, and @code{strcat}.
-
-@b{You can always determine the full length of a character's encoding
-from its first byte.} Guile provides the macro @code{scm_mb_len} which
-computes the encoding's length from its first byte. Given the first
-rule, you can see that @code{scm_mb_len (@var{b})}, for any @code{0 <=
-@var{b} <= 127}, returns 1.
-
-@b{Given an arbitrary byte position in a Guile string, you can always
-find the beginning and end of the character containing that byte without
-scanning too far in either direction.} This means that, if you are sure
-a byte sequence is a valid encoding of a character sequence, you can
-find character boundaries without keeping track of the beginning and
-ending of the overall string. This promise relies on the fact that, in
-addition to storing the string's length explicitly, Guile always either
-terminates the string's storage with a zero byte, or shares it with
-another string which is terminated this way.
-
-
-@node Functions for Operating on Multibyte Text, Multibyte Text Processing Errors, Promised Properties of the Guile Multibyte Encoding, Working With Multibyte Strings in C
-@section Functions for Operating on Multibyte Text
-
-Guile provides a variety of functions, variables, and types for working
-with multibyte text.
-
-@menu
-* Basic Multibyte Character Processing::
-* Finding Character Encoding Boundaries::
-* Multibyte String Functions::
-* Exchanging Guile Text With the Outside World in C::
-* Implementing Your Own Text Conversions::
-@end menu
-
-
-@node Basic Multibyte Character Processing, Finding Character Encoding Boundaries, Functions for Operating on Multibyte Text, Functions for Operating on Multibyte Text
-@subsection Basic Multibyte Character Processing
-
-Here are the essential types and functions for working with Guile text.
-Guile uses the C type @code{unsigned char *} to refer to text encoded
-with Guile's encoding.
-
-Note that any operation marked here as a ``Libguile Macro'' might
-evaluate its argument multiple times.
-
-@deftp {Libguile Type} scm_char_t
-This is a signed integral type large enough to hold any character in
-Guile's character set. All character numbers are positive.
-@end deftp
-
-@deftypefn {Libguile Macro} scm_char_t scm_mb_get (const unsigned char *@var{p})
-Return the character whose encoding starts at @var{p}. If @var{p} does
-not point at a valid character encoding, the behavior is undefined.
-@end deftypefn
-
-@deftypefn {Libguile Macro} int scm_mb_put (unsigned char *@var{p}, scm_char_t @var{c})
-Place the encoded form of the Guile character @var{c} at @var{p}, and
-return its length in bytes. If @var{c} is not a Guile character, the
-behavior is undefined.
-@end deftypefn
-
-@deftypevr {Libguile Constant} int scm_mb_max_len
-The maximum length of any character's encoding, in bytes. You may
-assume this is relatively small --- less than a dozen or so.
-@end deftypevr
-
-@deftypefn {Libguile Macro} int scm_mb_len (unsigned char @var{b})
-If @var{b} is the first byte of a character's encoding, return the full
-length of the character's encoding, in bytes. If @var{b} is not a valid
-leading byte, the behavior is undefined.
-@end deftypefn
-
-@deftypefn {Libguile Macro} int scm_mb_char_len (scm_char_t @var{c})
-Return the length of the encoding of the character @var{c}, in bytes.
-If @var{c} is not a valid Guile character, the behavior is undefined.
-@end deftypefn
-
-@deftypefn {Libguile Function} scm_char_t scm_mb_get_func (const unsigned char *@var{p})
-@deftypefnx {Libguile Function} int scm_mb_put_func (unsigned char *@var{p}, scm_char_t @var{c})
-@deftypefnx {Libguile Function} int scm_mb_len_func (unsigned char @var{b})
-@deftypefnx {Libguile Function} int scm_mb_char_len_func (scm_char_t @var{c})
-These are functions identical to the corresponding macros. You can use
-them in situations where the overhead of a function call is acceptable,
-and the cleaner semantics of function application are desireable.
-@end deftypefn
-
-
-@node Finding Character Encoding Boundaries, Multibyte String Functions, Basic Multibyte Character Processing, Functions for Operating on Multibyte Text
-@subsection Finding Character Encoding Boundaries
-
-These are functions for finding the boundaries between characters in
-multibyte text.
-
-Note that any operation marked here as a ``Libguile Macro'' might
-evaluate its argument multiple times, unless the definition promises
-otherwise.
-
-@deftypefn {Libguile Macro} int scm_mb_boundary_p (const unsigned char *@var{p})
-Return non-zero iff @var{p} points to the start of a character in
-multibyte text.
-
-This macro will evaluate its argument only once.
-@end deftypefn
-
-@deftypefn {Libguile Function} {const unsigned char *} scm_mb_floor (const unsigned char *@var{p})
-``Round'' @var{p} to the previous character boundary. That is, if
-@var{p} points to the middle of the encoding of a Guile character,
-return a pointer to the first byte of the encoding. If @var{p} points
-to the start of the encoding of a Guile character, return @var{p}
-unchanged.
-@end deftypefn
-
-@deftypefn {libguile Function} {const unsigned char *} scm_mb_ceiling (const unsigned char *@var{p})
-``Round'' @var{p} to the next character boundary. That is, if @var{p}
-points to the middle of the encoding of a Guile character, return a
-pointer to the first byte of the encoding of the next character. If
-@var{p} points to the start of the encoding of a Guile character, return
-@var{p} unchanged.
-@end deftypefn
-
-Note that it is usually not friendly for functions to silently correct
-byte offsets that point into the middle of a character's encoding. Such
-offsets almost always indicate a programming error, and they should be
-reported as early as possible. So, when you write code which operates
-on multibyte text, you should not use functions like these to ``clean
-up'' byte offsets which the originator believes to be correct; instead,
-your code should signal a @code{text:not-char-boundary} error as soon as
-it detects an invalid offset. @xref{Multibyte Text Processing Errors}.
-
-
-@node Multibyte String Functions, Exchanging Guile Text With the Outside World in C, Finding Character Encoding Boundaries, Functions for Operating on Multibyte Text
-@subsection Multibyte String Functions
-
-These functions allow you to operate on multibyte strings: sequences of
-character encodings.
-
-@deftypefn {Libguile Function} int scm_mb_count (const unsigned char *@var{p}, int @var{len})
-Return the number of Guile characters encoded by the @var{len} bytes at
-@var{p}.
-
-If the sequence contains any invalid character encodings, or ends with
-an incomplete character encoding, signal a @code{text:bad-encoding}
-error.
-@end deftypefn
-
-@deftypefn {Libguile Macro} scm_char_t scm_mb_walk (unsigned char **@var{pp})
-Return the character whose encoding starts at @code{*@var{pp}}, and
-advance @code{*@var{pp}} to the start of the next character. Return -1
-if @code{*@var{pp}} does not point to a valid character encoding.
-@end deftypefn
-
-@deftypefn {Libguile Function} {const unsigned char *} scm_mb_prev (const unsigned char *@var{p})
-If @var{p} points to the middle of the encoding of a Guile character,
-return a pointer to the first byte of the encoding. If @var{p} points
-to the start of the encoding of a Guile character, return the start of
-the previous character's encoding.
-
-This is like @code{scm_mb_floor}, but the returned pointer will always
-be before @var{p}. If you use this function to drive an iteration, it
-guarantees backward progress.
-@end deftypefn
-
-@deftypefn {Libguile Function} {const unsigned char *} scm_mb_next (const unsigned char *@var{p})
-If @var{p} points to the encoding of a Guile character, return a pointer
-to the first byte of the encoding of the next character.
-
-This is like @code{scm_mb_ceiling}, but the returned pointer will always
-be after @var{p}. If you use this function to drive an iteration, it
-guarantees forward progress.
-@end deftypefn
-
-@deftypefn {Libguile Function} {const unsigned char *} scm_mb_index (const unsigned char *@var{p}, int @var{len}, int @var{i})
-Assuming that the @var{len} bytes starting at @var{p} are a
-concatenation of valid character encodings, return a pointer to the
-start of the @var{i}'th character encoding in the sequence.
-
-This function scans the sequence from the beginning to find the
-@var{i}'th character, and will generally require time proportional to
-the distance from @var{p} to the returned address.
-
-If the sequence contains any invalid character encodings, or ends with
-an incomplete character encoding, signal a @code{text:bad-encoding}
-error.
-@end deftypefn
-
-It is common to process the characters in a string from left to right.
-However, if you fetch each character using @code{scm_mb_index}, each
-call will scan the text from the beginning, so your loop will require
-time proportional to at least the square of the length of the text. To
-avoid this poor performance, you can use an @code{scm_mb_cache}
-structure and the @code{scm_mb_index_cached} macro.
-
-@deftp {Libguile Type} {struct scm_mb_cache}
-This structure holds information that allows a string scanning operation
-to use the results from a previous scan of the string. It has the
-following members:
-@table @code
-
-@item character
-An index, in characters, into the string.
-
-@item byte
-The index, in bytes, of the start of that character.
-
-@end table
-
-In other words, @code{byte} is the byte offset of the
-@code{character}'th character of the string. Note that if @code{byte}
-and @code{character} are equal, then all characters before that point
-must have encodings exactly one byte long, and the string can be indexed
-normally.
-
-All elements of a @code{struct scm_mb_cache} structure should be
-initialized to zero before its first use, and whenever the string's text
-changes.
-@end deftp
-
-@deftypefn {Libguile Macro} const unsigned char *scm_mb_index_cached (const unsigned char *@var{p}, int @var{len}, int @var{i}, struct scm_mb_cache *@var{cache})
-@deftypefnx {Libguile Function} const unsigned char *scm_mb_index_cached_func (const unsigned char *@var{p}, int @var{len}, int @var{i}, struct scm_mb_cache *@var{cache})
-This macro and this function are identical to @code{scm_mb_index},
-except that they may consult and update *@var{cache} in order to avoid
-scanning the string from the beginning. @code{scm_mb_index_cached} is a
-macro, so it may have less overhead than
-@code{scm_mb_index_cached_func}, but it may evaluate its arguments more
-than once.
-
-Using @code{scm_mb_index_cached} or @code{scm_mb_index_cached_func}, you
-can scan a string from left to right, or from right to left, in time
-proportional to the length of the string. As long as each character
-fetched is less than some constant distance before or after the previous
-character fetched with @var{cache}, each access will require constant
-time.
-@end deftypefn
-
-Guile also provides functions to convert between an encoded sequence of
-characters, and an array of @code{scm_char_t} objects.
-
-@deftypefn {Libguile Function} scm_char_t *scm_mb_multibyte_to_fixed (const unsigned char *@var{p}, int @var{len}, int *@var{result_len})
-Convert the variable-width text in the @var{len} bytes at @var{p}
-to an array of @code{scm_char_t} values. Return a pointer to the array,
-and set @code{*@var{result_len}} to the number of elements it contains.
-The returned array is allocated with @code{malloc}, and it is the
-caller's responsibility to free it.
-
-If the text is not a sequence of valid character encodings, this
-function will signal a @code{text:bad-encoding} error.
-@end deftypefn
-
-@deftypefn {Libguile Function} unsigned char *scm_mb_fixed_to_multibyte (const scm_char_t *@var{fixed}, int @var{len}, int *@var{result_len})
-Convert the array of @code{scm_char_t} values to a sequence of
-variable-width character encodings. Return a pointer to the array of
-bytes, and set @code{*@var{result_len}} to its length, in bytes.
-
-The returned byte sequence is terminated with a zero byte, which is not
-counted in the length returned in @code{*@var{result_len}}.
-
-The returned byte sequence is allocated with @code{malloc}; it is the
-caller's responsibility to free it.
-
-If the text is not a sequence of valid character encodings, this
-function will signal a @code{text:bad-encoding} error.
-@end deftypefn
-
-
-@node Exchanging Guile Text With the Outside World in C, Implementing Your Own Text Conversions, Multibyte String Functions, Functions for Operating on Multibyte Text
-@subsection Exchanging Guile Text With the Outside World in C
-
-[[This is kind of a heavy-weight model, given that one end of the
-conversion is always going to be the Guile encoding. Any way to shorten
-things a bit?]]
-
-Guile provides functions for converting between Guile's internal text
-representation and encodings popular in the outside world. These
-functions are closely modeled after the @code{iconv} functions available
-on some systems.
-
-To convert text between two encodings, you should first call
-@code{scm_mb_iconv_open} to indicate the source and destination
-encodings; this function returns a context object which records the
-conversion to perform.
-
-Then, you should call @code{scm_mb_iconv} to actually convert the text.
-This function expects input and output buffers, and a pointer to the
-context you got from @var{scm_mb_iconv_open}. You don't need to pass
-all your input to @code{scm_mb_iconv} at once; you can invoke it on
-successive blocks of input (as you read it from a file, say), and it
-will convert as much as it can each time, indicating when you should
-grow your output buffer.
-
-An encoding may be @dfn{stateless}, or @dfn{stateful}. In most
-encodings, a contiguous group of bytes from the sequence completely
-specifies a particular character; these are stateless encodings.
-However, some encodings require you to look back an unbounded number of
-bytes in the stream to assign a meaning to a particular byte sequence;
-such encodings are stateful.
-
-For example, in the @samp{ISO-2022-JP} encoding for Japanese text, the
-byte sequence @samp{27 36 66} indicates that subsequent bytes should be
-taken in pairs and interpreted as characters from the JIS-0208 character
-set. An arbitrary number of byte pairs may follow this sequence. The
-byte sequence @samp{27 40 66} indicates that subsequent bytes should be
-interpreted as @sc{ASCII}. In this encoding, you cannot tell whether a
-given byte is an @sc{ASCII} character without looking back an arbitrary
-distance for the most recent escape sequence, so it is a stateful
-encoding.
-
-In Guile, if a conversion involves a stateful encoding, the context
-object carries any necessary state. Thus, you can have many independent
-conversions to or from stateful encodings taking place simultaneously,
-as long as each data stream uses its own context object for the
-conversion.
-
-@deftp {Libguile Type} {struct scm_mb_iconv}
-This is the type for context objects, which represent the encodings and
-current state of an ongoing text conversion. A @code{struct
-scm_mb_iconv} records the source and destination encodings, and keeps
-track of any information needed to handle stateful encodings.
-@end deftp
-
-@deftypefn {Libguile Function} {struct scm_mb_iconv *} scm_mb_iconv_open (const char *@var{tocode}, const char *@var{fromcode})
-Return a pointer to a new @code{struct scm_mb_iconv} context object,
-ready to convert from the encoding named @var{fromcode} to the encoding
-named @var{tocode}. For stateful encodings, the context object is in
-some appropriate initial state, ready for use with the
-@code{scm_mb_iconv} function.
-
-When you are done using a context object, you may call
-@code{scm_mb_iconv_close} to free it.
-
-If either @var{tocode} or @var{fromcode} is not the name of a known
-encoding, this function will signal the @code{text:unknown-conversion}
-error, described below.
-
-@c Try to use names here from the IANA list:
-@c see ftp://ftp.isi.edu/in-notes/iana/assignments/character-sets
-Guile supports at least these encodings:
-@table @samp
-
-@item US-ASCII
-@sc{US-ASCII}, in the standard one-character-per-byte encoding.
-
-@item ISO-8859-1
-The usual character set for Western European languages, in its usual
-one-character-per-byte encoding.
-
-@item Guile-MB
-Guile's current internal multibyte encoding. The actual encoding this
-name refers to will change from one version of Guile to the next. You
-should use this when converting data between external sources and the
-encoding used by Guile objects.
-
-You should @emph{not} use this as the encoding for data presented to the
-outside world, for two reasons. 1) Its meaning will change over time,
-so data written using the @samp{guile} encoding with one version of
-Guile might not be readable with the @samp{guile} encoding in another
-version of Guile. 2) It currently corresponds to @samp{Emacs-Mule},
-which invented for Emacs's internal use, and was never intended to serve
-as an exchange medium.
-
-@item Guile-Wide
-Guile's character set, as an array of @code{scm_char_t} values.
-
-Note that this encoding is even less suitable for public use than
-@samp{Guile}, since the exact sequence of bytes depends heavily on the
-size and endianness the host system uses for @code{scm_char_t}. Using
-this encoding is very much like calling the
-@code{scm_mb_multibyte_to_fixed} or @code{scm_mb_fixed_to_multibyte}
-functions, except that @code{scm_mb_iconv} gives you more control over
-buffer allocation and management.
-
-@item Emacs-Mule
-This is the variable-length encoding for multi-lingual text by GNU
-Emacs, at least through version 20.4. You probably should not use this
-encoding, as it is designed only for Emacs's internal use. However, we
-provide it here because it's trivial to support, and some people
-probably do have @samp{emacs-mule}-format files lying around.
-
-@end table
-
-(At the moment, this list doesn't include any character sets suitable for
-external use that can actually handle multilingual data; this is
-unfortunate, as it encourages users to write data in Emacs-Mule format,
-which nobody but Emacs and Guile understands. We hope to add support
-for Unicode in UTF-8 soon, which should solve this problem.)
-
-Case is not significant in encoding names.
-
-You can define your own conversions; see @ref{Implementing Your Own Text
-Conversions}.
-@end deftypefn
-
-@deftypefn {Libguile Function} int scm_mb_have_encoding (const char *@var{encoding})
-Return a non-zero value if Guile supports the encoding named @var{encoding}[[]]
-@end deftypefn
-
-@deftypefn {Libguile Function} size_t scm_mb_iconv (struct scm_mb_iconv *@var{context}, const char **@var{inbuf}, size_t *@var{inbytesleft}, char **@var{outbuf}, size_t *@var{outbytesleft})
-Convert a sequence of characters from one encoding to another. The
-argument @var{context} specifies the encodings to use for the input and
-output, and carries state for stateful encodings; use
-@code{scm_mb_iconv_open} to create a @var{context} object for a
-particular conversion.
-
-Upon entry to the function, @code{*@var{inbuf}} should point to the
-input buffer, and @code{*@var{inbytesleft}} should hold the number of
-input bytes present in the buffer; @code{*@var{outbuf}} should point to
-the output buffer, and @code{*@var{outbytesleft}} should hold the number
-of bytes available to hold the conversion results in that buffer.
-
-Upon exit from the function, @code{*@var{inbuf}} points to the first
-unconsumed byte of input, and @code{*@var{inbytesleft}} holds the number
-of unconsumed input bytes; @code{*@var{outbuf}} points to the byte after
-the last output byte, and @code{*@var{outbyteleft}} holds the number of
-bytes left unused in the output buffer.
-
-For stateful encodings, @var{context} carries encoding state from one
-call to @code{scm_mb_iconv} to the next. Thus, successive calls to
-@var{scm_mb_iconv} which use the same context object can convert a
-stream of data one chunk at a time.
-
-If @var{inbuf} is zero or @code{*@var{inbuf}} is zero, then the call is
-taken as a request to reset the states of the input and the output
-encodings. If @var{outbuf} is non-zero and @code{*@var{outbuf}} is
-non-zero, then @code{scm_mb_iconv} stores a byte sequence in the output
-buffer to put the output encoding in its initial state. If the output
-buffer is not large enough to hold this byte sequence,
-@code{scm_mb_iconv} returns @code{scm_mb_iconv_too_big}, and leaves
-the shift states of @var{context}'s input and output encodings
-unchanged.
-
-The @code{scm_mb_iconv} function always consumes only complete
-characters or shift sequences from the input buffer, and the output
-buffer always contains a sequence of complete characters or escape
-sequences.
-
-If the input sequence contains characters which are not expressible in
-the output encoding, @code{scm_mb_iconv} converts it in an
-implementation-defined way. It may simply delete the character.
-
-Some encodings use byte sequences which do not correspond to any textual
-character. For example, the escape sequence of a stateful encoding has
-no textual meaning. When converting from such an encoding, a call to
-@code{scm_mb_iconv} might consume input but produce no output, since the
-input sequence might contain only escape sequences.
-
-Normally, @code{scm_mb_iconv} returns the number of input characters it
-could not convert perfectly to the ouput encoding. However, it may
-return one of the @code{scm_mb_iconv_} codes described below, to
-indicate an error. All of these codes are negative values.
-
-If the input sequence contains an invalid character encoding, conversion
-stops before the invalid input character, and @code{scm_mb_iconv}
-returns the constant value @code{scm_mb_iconv_bad_encoding}.
-
-If the input sequence ends with an incomplete character encoding,
-@code{scm_mb_iconv} will leave it in the input buffer, unconsumed, and
-return the constant value @code{scm_mb_iconv_incomplete_encoding}. This
-is not necessarily an error, if you expect to call @code{scm_mb_iconv}
-again with more data which might contain the rest of the encoding
-fragment.
-
-If the output buffer does not contain enough room to hold the converted
-form of the complete input text, @code{scm_mb_iconv} converts as much as
-it can, changes the input and output pointers to reflect the amount of
-text successfully converted, and then returns
-@code{scm_mb_iconv_too_big}.
-@end deftypefn
-
-Here are the status codes that might be returned by @code{scm_mb_iconv}.
-They are all negative integers.
-@table @code
-
-@item scm_mb_iconv_too_big
-The conversion needs more room in the output buffer. Some characters
-may have been consumed from the input buffer, and some characters may
-have been placed in the available space in the output buffer.
-
-@item scm_mb_iconv_bad_encoding
-@code{scm_mb_iconv} encountered an invalid character encoding in the
-input buffer. Conversion stopped before the invalid character, so there
-may be some characters consumed from the input buffer, and some
-converted text in the output buffer.
-
-@item scm_mb_iconv_incomplete_encoding
-The input buffer ends with an incomplete character encoding. The
-incomplete encoding is left in the input buffer, unconsumed. This is
-not necessarily an error, if you expect to call @code{scm_mb_iconv}
-again with more data which might contain the rest of the incomplete
-encoding.
-
-@end table
-
-
-Finally, Guile provides a function for destroying conversion contexts.
-
-@deftypefn {Libguile Function} void scm_mb_iconv_close (struct scm_mb_iconv *@var{context})
-Deallocate the conversion context object @var{context}, and all other
-resources allocated by the call to @code{scm_mb_iconv_open} which
-returned @var{context}.
-@end deftypefn
-
-
-@node Implementing Your Own Text Conversions, , Exchanging Guile Text With the Outside World in C, Functions for Operating on Multibyte Text
-@subsection Implementing Your Own Text Conversions
-
-[[note that conversions to and from Guile must produce streams
-containing only valid character encodings, or else Guile will crash]]
-
-This section describes the interface for adding your own encoding
-conversions for use with @code{scm_mb_iconv}. The interface here is
-borrowed from the GNOME Project's @file{libunicode} library.
-
-Guile's @code{scm_mb_iconv} function works by converting the input text
-to a stream of @code{scm_char_t} characters, and then converting
-those characters to the desired output encoding. This makes it easy
-for Guile to choose the appropriate conversion back ends for an
-arbitrary pair of input and output encodings, but it also means that the
-accuracy and quality of the conversions depends on the fidelity of
-Guile's internal character set to the source and destination encodings.
-Since @code{scm_mb_iconv} will be used almost exclusively for converting
-to and from Guile's internal character set, this shouldn't be a problem.
-
-To add support for a particular encoding to Guile, you must provide one
-function (called the @dfn{read} function) which converts from your
-encoding to an array of @code{scm_char_t}'s, and another function
-(called the @dfn{write} function) to convert from an array of
-@code{scm_char_t}'s back into your encoding. To convert from some
-encoding @var{a} to some other encoding @var{b}, Guile pairs up
-@var{a}'s read function with @var{b}'s write function. Each call to
-@code{scm_mb_iconv} passes text in encoding @var{a} through the read
-function, to produce an array of @code{scm_char_t}'s, and then passes
-that array to the write function, to produce text in encoding @var{b}.
-
-For stateful encodings, a read or write function can hang its own data
-structures off the conversion object, and provide its own functions to
-allocate and destroy them; this allows read and write functions to
-maintain whatever state they like.
-
-The Guile conversion back end represents each available encoding with a
-@code{struct scm_mb_encoding} object.
-
-@deftp {Libguile Type} {struct scm_mb_encoding}
-This data structure describes an encoding. It has the following
-members:
-
-@table @code
-
-@item char **names
-An array of strings, giving the various names for this encoding. The
-array should be terminated by a zero pointer. Case is not significant
-in encoding names.
-
-The @code{scm_mb_iconv_open} function searches the list of registered
-encodings for an encoding whose @code{names} array matches its
-@var{tocode} or @var{fromcode} argument.
-
-@item int (*init) (void **@var{cookie})
-An initialization function for the encoding's private data.
-@code{scm_mb_iconv_open} will call this function, passing it the address
-of the cookie for this encoding in this context. (We explain cookies
-below.) There is no way for the @code{init} function to tell whether
-the encoding will be used for reading or writing.
-
-Note that @code{init} receives a @emph{pointer} to the cookie, not the
-cookie itself. Because the type of @var{cookie} is @code{void **}, the
-C compiler will not check it as carefully as it would other types.
-
-The @code{init} member may be zero, indicating that no initialization is
-necessary for this encoding.
-
-@item int (*destroy) (void **@var{cookie})
-A deallocation function for the encoding's private data.
-@code{scm_mb_iconv_close} calls this function, passing it the address of
-the cookie for this encoding in this context. The @code{destroy}
-function should free any data the @code{init} function allocated.
-
-Note that @code{destroy} receives a @emph{pointer} to the cookie, not the
-cookie itself. Because the type of @var{cookie} is @code{void **}, the
-C compiler will not check it as carefully as it would other types.
-
-The @code{destroy} member may be zero, indicating that this encoding
-doesn't need to perform any special action to destroy its local data.
-
-@item int (*reset) (void *@var{cookie}, char **@var{outbuf}, size_t *@var{outbytesleft})
-Put the encoding into its initial shift state. Guile calls this
-function whether the encoding is being used for input or output, so this
-should take appropriate steps for both directions. If @var{outbuf} and
-@var{outbytesleft} are valid, the reset function should emit an escape
-sequence to reset the output stream to its initial state; @var{outbuf}
-and @var{outbytesleft} should be handled just as for
-@code{scm_mb_iconv}.
-
-This function can return an @code{scm_mb_iconv_} error code
-(@pxref{Exchanging Guile Text With the Outside World in C}). If it
-returns @code{scm_mb_iconv_too_big}, then the output buffer's shift
-state must be left unchanged.
-
-Note that @code{reset} receives the cookie's value itself, not a pointer
-to the cookie, as the @code{init} and @code{destroy} functions do.
-
-The @code{reset} member may be zero, indicating that this encoding
-doesn't use a shift state.
-
-@item enum scm_mb_read_result (*read) (void *@var{cookie}, const char **@var{inbuf}, size_t *@var{inbytesleft}, scm_char_t **@var{outbuf}, size_t *@var{outcharsleft})
-Read some bytes and convert into an array of Guile characters. This is
-the encoding's read function.
-
-On entry, there are *@var{inbytesleft} bytes of text at *@var{inbuf} to
-be converted, and *@var{outcharsleft} characters available at
-*@var{outbuf} to hold the results.
-
-On exit, *@var{inbytesleft} and *@var{inbuf} indicate the input bytes
-still not consumed. *@var{outcharsleft} and *@var{outbuf} indicate the
-output buffer space still not filled. (By exclusion, these indicate
-which input bytes were consumed, and which output characters were
-produced.)
-
-Return one of the @code{enum scm_mb_read_result} values, described below.
-
-Note that @code{read} receives the cookie's value itself, not a pointer
-to the cookie, as the @code{init} and @code{destroy} functions do.
-
-@item enum scm_mb_write_result (*write) (void *@var{cookie}, scm_char_t **@var{inbuf}, size_t *@var{incharsleft}, **@var{outbuf}, size_t *@var{outbytesleft})
-Convert an array of Guile characters to output bytes. This is
-the encoding's write function.
-
-On entry, there are *@var{incharsleft} Guile characters available at
-*@var{inbuf}, and *@var{outbytesleft} bytes available to store output at
-*@var{outbuf}.
-
-On exit, *@var{incharsleft} and *@var{inbuf} indicate the number of
-Guile characters left unconverted (because there was insufficient room
-in the output buffer to hold their converted forms), and
-*@var{outbytesleft} and *@var{outbuf} indicate the unused portion of the
-output buffer.
-
-Return one of the @code{scm_mb_write_result} values, described below.
-
-Note that @code{write} receives the cookie's value itself, not a pointer
-to the cookie, as the @code{init} and @code{destroy} functions do.
-
-@item struct scm_mb_encoding *next
-This is used by Guile to maintain a linked list of encodings. It is
-filled in when you call @code{scm_mb_register_encoding} to add your
-encoding to the list.
-
-@end table
-@end deftp
-
-Here is the enumerated type for the values an encoding's read function
-can return:
-
-@deftp {Libguile Type} {enum scm_mb_read_result}
-This type represents the result of a call to an encoding's read
-function. It has the following values:
-
-@table @code
-
-@item scm_mb_read_ok
-The read function consumed at least one byte of input.
-
-@item scm_mb_read_incomplete
-The data present in the input buffer does not contain a complete
-character encoding. No input was consumed, and no characters were
-produced as output. This is not necessarily an error status, if there
-is more data to pass through.
-
-@item scm_mb_read_error
-The input contains an invalid character encoding.
-
-@end table
-@end deftp
-
-Here is the enumerated type for the values an encoding's write function
-can return:
-
-@deftp {Libguile Type} {enum scm_mb_write_result}
-This type represents the result of a call to an encoding's write
-function. It has the following values:
-
-@table @code
-
-@item scm_mb_write_ok
-The write function was able to convert all the characters in @var{inbuf}
-successfully.
-
-@item scm_mb_write_too_big
-The write function filled the output buffer, but there are still
-characters in @var{inbuf} left unconsumed; @var{inbuf} and
-@var{incharsleft} indicate the unconsumed portion of the input buffer.
-
-@end table
-@end deftp
-
-
-Conversions to or from stateful encodings need to keep track of each
-encoding's current state. Each conversion context contains two
-@code{void *} variables called @dfn{cookies}, one for the input
-encoding, and one for the output encoding. These cookies are passed to
-the encodings' functions, for them to use however they please. A
-stateful encoding can use its cookie to hold a pointer to some object
-which maintains the context's current shift state. Stateless encodings
-will probably not use their cookies.
-
-The cookies' lifetime is the same as that of the context object. When
-the user calls @code{scm_mb_iconv_close} to destroy a context object,
-@code{scm_mb_iconv_close} calls the input and output encodings'
-@code{destroy} functions, passing them their respective cookies, so each
-encoding can free any data it allocated for that context.
-
-Note that, if a read or write function returns a successful result code
-like @code{scm_mb_read_ok} or @code{scm_mb_write_ok}, then the remaining
-input, together with the output, must together represent the complete
-input text; the encoding may not store any text temporarily in its
-cookie. This is because, if @code{scm_mb_iconv} returns a successful
-result to the user, it is correct for the user to assume that all the
-consumed input has been converted and placed in the output buffer.
-There is no ``flush'' operation to push any final results out of the
-encodings' buffers.
-
-Here is the function you call to register a new encoding with the
-conversion system:
-
-@deftypefn {Libguile Function} void scm_mb_register_encoding (struct scm_mb_encoding *@var{encoding})
-Add the encoding described by @code{*@var{encoding}} to the set
-understood by @code{scm_mb_iconv_open}. Once you have registered your
-encoding, you can use it by calling @code{scm_mb_iconv_open} with one of
-the names in @code{@var{encoding}->names}.
-@end deftypefn
-
-
-@node Multibyte Text Processing Errors, Why Guile Does Not Use a Fixed-Width Encoding, Functions for Operating on Multibyte Text, Working With Multibyte Strings in C
-@section Multibyte Text Processing Errors
-
-This section describes error conditions which code can signal to
-indicate problems encountered while processing multibyte text. In each
-case, the arguments @var{message} and @var{args} are an error format
-string and arguments to be substituted into the string, as accepted by
-the @code{display-error} function.
-
-@deffn Condition text:not-char-boundary func message args object offset
-By calling @var{func}, the program attempted to access a character at
-byte offset @var{offset} in the Guile object @var{object}, but
-@var{offset} is not the start of a character's encoding in @var{object}.
-
-Typically, @var{object} is a string or symbol. If the function signalling
-the error cannot find the Guile object that contains the text it is
-inspecting, it should use @code{#f} for @var{object}.
-@end deffn
-
-@deffn Condition text:bad-encoding func message args object
-By calling @var{func}, the program attempted to interpret the text in
-@var{object}, but @var{object} contains a byte sequence which is not a
-valid encoding for any character.
-@end deffn
-
-@deffn Condition text:not-guile-char func message args number
-By calling @var{func}, the program attempted to treat @var{number} as the
-number of a character in the Guile character set, but @var{number} does
-not correspond to any character in the Guile character set.
-@end deffn
-
-@deffn Condition text:unknown-conversion func message args from to
-By calling @var{func}, the program attempted to convert from an encoding
-named @var{from} to an encoding named @var{to}, but Guile does not
-support such a conversion.
-@end deffn
-
-@deftypevr {Libguile Variable} SCM scm_text_not_char_boundary
-@deftypevrx {Libguile Variable} SCM scm_text_bad_encoding
-@deftypevrx {Libguile Variable} SCM scm_text_not_guile_char
-These variables hold the scheme symbol objects whose names are the
-condition symbols above. You can use these when signalling these
-errors, instead of looking them up yourself.
-@end deftypevr
-
-
-@node Why Guile Does Not Use a Fixed-Width Encoding, , Multibyte Text Processing Errors, Working With Multibyte Strings in C
-@section Why Guile Does Not Use a Fixed-Width Encoding
-
-Multibyte encodings are clumsier to work with than encodings which use a
-fixed number of bytes for every character. For example, using a
-fixed-width encoding, we can extract the @var{i}th character of a string
-in constant time, and we can always substitute the @var{i}th character
-of a string with any other character without reallocating or copying the
-string.
-
-However, there are no fixed-width encodings which include the characters
-we wish to include, and also fit in a reasonable amount of space.
-Despite the Unicode standard's claims to the contrary, Unicode is not
-really a fixed-width encoding. Unicode uses surrogate pairs to
-represent characters outside the 16-bit range; a surrogate pair must be
-treated as a single character, but occupies two 16-bit spaces. As of
-this writing, there are already plans to assign characters to the
-surrogate character codes. Three- and four-byte encodings are
-too wasteful for a majority of Guile's users, who only need @sc{ASCII}
-and a few accented characters.
-
-Another alternative would be to have several different fixed-width
-string representations, each with a different element size. For each
-string, Guile would use the smallest element size capable of
-accomodating the string's text. This would allow users of English and
-the Western European languages to use the traditional memory-efficient
-encodings. However, if Guile has @var{n} string representations, then
-users must write @var{n} versions of any code which manipulates text
-directly --- one for each element size. And if a user wants to operate
-on two strings simultaneously, and wants to avoid testing the string
-sizes within the loop, she must make @var{n}*@var{n} copies of the loop.
-Most users will simply not bother. Instead, they will write code which
-supports only one string size, leaving us back where we started. By
-using a single internal representation, Guile makes it easier for users
-to write multilingual code.
-
-[[What about tagging each string with its encoding?
-"Every extension must be written to deal with every encoding"]]
-
-[[You don't really want to index strings anyway.]]
-
-Finally, Guile's multibyte encoding is not so bad. Unlike a two- or
-four-byte encoding, it is efficient in space for American and European
-users. Furthermore, the properties described above mean that many
-functions can be coded just as they would for a single-byte encoding;
-see @ref{Promised Properties of the Guile Multibyte Encoding}.
-
-@bye
HCoop / bpt / guile.git / commitdiff