/*** 0. General comments ***/
-/*** GENERAL NOTE on CODING SYSTEM ***
+/*** GENERAL NOTE on CODING SYSTEMS ***
- Coding system is an encoding mechanism of one or more character
+ A coding system is an encoding mechanism for one or more character
sets. Here's a list of coding systems which Emacs can handle. When
we say "decode", it means converting some other coding system to
- Emacs' internal format (emacs-internal), and when we say "encode",
+ Emacs' internal format (emacs-mule), and when we say "encode",
it means converting the coding system emacs-mule to some other
coding system.
0. Emacs' internal format (emacs-mule)
- Emacs itself holds a multi-lingual character in a buffer and a string
+ Emacs itself holds a multi-lingual character in buffers and strings
in a special format. Details are described in section 2.
1. ISO2022
3. BIG5
- A coding system to encode character sets: ASCII and Big5. Widely
- used by Chinese (mainly in Taiwan and Hong Kong). Details are
+ A coding system to encode the character sets ASCII and Big5. Widely
+ used for Chinese (mainly in Taiwan and Hong Kong). Details are
described in section 4. In this file, when we write "BIG5"
(all uppercase), we mean the coding system, and when we write
"Big5" (capitalized), we mean the character set.
4. Raw text
- A coding system for a text containing random 8-bit code. Emacs does
- no code conversion on such a text except for end-of-line format.
+ A coding system for text containing random 8-bit code. Emacs does
+ no code conversion on such text except for end-of-line format.
5. Other
- If a user wants to read/write a text encoded in a coding system not
- listed above, he can supply a decoder and an encoder for it in CCL
+ If a user wants to read/write text encoded in a coding system not
+ listed above, he can supply a decoder and an encoder for it as CCL
(Code Conversion Language) programs. Emacs executes the CCL program
while reading/writing.
/*** GENERAL NOTES on END-OF-LINE FORMAT ***
- How end-of-line of a text is encoded depends on a system. For
- instance, Unix's format is just one byte of `line-feed' code,
+ How end-of-line of text is encoded depends on the operating system.
+ For instance, Unix's format is just one byte of `line-feed' code,
whereas DOS's format is two-byte sequence of `carriage-return' and
`line-feed' codes. MacOS's format is usually one byte of
`carriage-return'.
- Since text characters encoding and end-of-line encoding are
- independent, any coding system described above can take
- any format of end-of-line. So, Emacs has information of format of
- end-of-line in each coding-system. See section 6 for more details.
+ Since text character encoding and end-of-line encoding are
+ independent, any coding system described above can have any
+ end-of-line format. So Emacs has information about end-of-line
+ format in each coding-system. See section 6 for more details.
*/
These functions check if a text between SRC and SRC_END is encoded
in the coding system category XXX. Each returns an integer value in
- which appropriate flag bits for the category XXX is set. The flag
+ which appropriate flag bits for the category XXX are set. The flag
bits are defined in macros CODING_CATEGORY_MASK_XXX. Below is the
- template of these functions. */
+ template for these functions. If MULTIBYTEP is nonzero, 8-bit codes
+ of the range 0x80..0x9F are in multibyte form. */
#if 0
int
-detect_coding_emacs_mule (src, src_end)
+detect_coding_emacs_mule (src, src_end, multibytep)
unsigned char *src, *src_end;
+ int multibytep;
{
...
}
multibyte text goes to a place pointed to by DESTINATION, the length
of which should not exceed DST_BYTES.
- These functions set the information of original and decoded texts in
- the members produced, produced_char, consumed, and consumed_char of
- the structure *CODING. They also set the member result to one of
- CODING_FINISH_XXX indicating how the decoding finished.
+ These functions set the information about original and decoded texts
+ in the members `produced', `produced_char', `consumed', and
+ `consumed_char' of the structure *CODING. They also set the member
+ `result' to one of CODING_FINISH_XXX indicating how the decoding
+ finished.
- DST_BYTES zero means that source area and destination area are
+ DST_BYTES zero means that the source area and destination area are
overlapped, which means that we can produce a decoded text until it
- reaches at the head of not-yet-decoded source text.
+ reaches the head of the not-yet-decoded source text.
- Below is a template of these functions. */
+ Below is a template for these functions. */
#if 0
static void
decode_coding_XXX (coding, source, destination, src_bytes, dst_bytes)
/*** GENERAL NOTES on `encode_coding_XXX ()' functions ***
- These functions encode SRC_BYTES length text at SOURCE of Emacs'
+ These functions encode SRC_BYTES length text at SOURCE from Emacs'
internal multibyte format to CODING. The resulting unibyte text
goes to a place pointed to by DESTINATION, the length of which
should not exceed DST_BYTES.
- These functions set the information of original and encoded texts in
- the members produced, produced_char, consumed, and consumed_char of
- the structure *CODING. They also set the member result to one of
- CODING_FINISH_XXX indicating how the encoding finished.
+ These functions set the information about original and encoded texts
+ in the members `produced', `produced_char', `consumed', and
+ `consumed_char' of the structure *CODING. They also set the member
+ `result' to one of CODING_FINISH_XXX indicating how the encoding
+ finished.
- DST_BYTES zero means that source area and destination area are
- overlapped, which means that we can produce a encoded text until it
- reaches at the head of not-yet-encoded source text.
+ DST_BYTES zero means that the source area and destination area are
+ overlapped, which means that we can produce encoded text until it
+ reaches at the head of the not-yet-encoded source text.
- Below is a template of these functions. */
+ Below is a template for these functions. */
#if 0
static void
encode_coding_XXX (coding, source, destination, src_bytes, dst_bytes)
} while (0)
+/* Like ONE_MORE_BYTE, but 8-bit bytes of data at SRC are in multibyte
+ form if MULTIBYTEP is nonzero. */
+
+#define ONE_MORE_BYTE_CHECK_MULTIBYTE(c1, multibytep) \
+ do { \
+ if (src >= src_end) \
+ { \
+ coding->result = CODING_FINISH_INSUFFICIENT_SRC; \
+ goto label_end_of_loop; \
+ } \
+ c1 = *src++; \
+ if (multibytep && c1 == LEADING_CODE_8_BIT_CONTROL) \
+ c1 = *src++ - 0x20; \
+ } while (0)
+
/* Set C to the next character at the source text pointed by `src'.
If there are not enough characters in the source, jump to
`label_end_of_loop'. The caller should set variables `coding'
} while (0)
-/* Produce a multibyte form of characater C to `dst'. Jump to
+/* Produce a multibyte form of character C to `dst'. Jump to
`label_end_of_loop' if there's not enough space at `dst'.
- If we are now in the middle of composition sequence, the decoded
+ If we are now in the middle of a composition sequence, the decoded
character may be ALTCHAR (for the current composition). In that
case, the character goes to coding->cmp_data->data instead of
`dst'.
struct coding_system *coding_system_table[CODING_CATEGORY_IDX_MAX];
/* Table of coding category masks. Nth element is a mask for a coding
- cateogry of which priority is Nth. */
+ category of which priority is Nth. */
static
int coding_priorities[CODING_CATEGORY_IDX_MAX];
\f
/*** 2. Emacs internal format (emacs-mule) handlers ***/
-/* Emacs' internal format for encoding multiple character sets is a
- kind of multi-byte encoding, i.e. characters are encoded by
- variable-length sequences of one-byte codes.
+/* Emacs' internal format for representation of multiple character
+ sets is a kind of multi-byte encoding, i.e. characters are
+ represented by variable-length sequences of one-byte codes.
ASCII characters and control characters (e.g. `tab', `newline') are
represented by one-byte sequences which are their ASCII codes, in
The other characters are represented by a sequence of `base
leading-code', optional `extended leading-code', and one or two
`position-code's. The length of the sequence is determined by the
- base leading-code. Leading-code takes the range 0x80 through 0x9F,
+ base leading-code. Leading-code takes the range 0x81 through 0x9D,
whereas extended leading-code and position-code take the range 0xA0
through 0xFF. See `charset.h' for more details about leading-code
and position-code.
ascii 0x00..0x7F
eight-bit-control LEADING_CODE_8_BIT_CONTROL + 0xA0..0xBF
eight-bit-graphic 0xA0..0xBF
- ELSE 0x81..0x9F + [0xA0..0xFF]+
+ ELSE 0x81..0x9D + [0xA0..0xFF]+
---------------------------------------------
+ As this is the internal character representation, the format is
+ usually not used externally (i.e. in a file or in a data sent to a
+ process). But, it is possible to have a text externally in this
+ format (i.e. by encoding by the coding system `emacs-mule').
+
+ In that case, a sequence of one-byte codes has a slightly different
+ form.
+
+ Firstly, all characters in eight-bit-control are represented by
+ one-byte sequences which are their 8-bit code.
+
+ Next, character composition data are represented by the byte
+ sequence of the form: 0x80 METHOD BYTES CHARS COMPONENT ...,
+ where,
+ METHOD is 0xF0 plus one of composition method (enum
+ composition_method),
+
+ BYTES is 0xA0 plus the byte length of these composition data,
+
+ CHARS is 0xA0 plus the number of characters composed by these
+ data,
+
+ COMPONENTs are characters of multibyte form or composition
+ rules encoded by two-byte of ASCII codes.
+
+ In addition, for backward compatibility, the following formats are
+ also recognized as composition data on decoding.
+
+ 0x80 MSEQ ...
+ 0x80 0xFF MSEQ RULE MSEQ RULE ... MSEQ
+
+ Here,
+ MSEQ is a multibyte form but in these special format:
+ ASCII: 0xA0 ASCII_CODE+0x80,
+ other: LEADING_CODE+0x20 FOLLOWING-BYTE ...,
+ RULE is a one byte code of the range 0xA0..0xF0 that
+ represents a composition rule.
*/
enum emacs_code_class_type emacs_code_class[256];
Check if a text is encoded in Emacs' internal format. If it is,
return CODING_CATEGORY_MASK_EMACS_MULE, else return 0. */
-int
-detect_coding_emacs_mule (src, src_end)
+static int
+detect_coding_emacs_mule (src, src_end, multibytep)
unsigned char *src, *src_end;
+ int multibytep;
{
unsigned char c;
int composing = 0;
while (1)
{
- ONE_MORE_BYTE (c);
+ ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
if (composing)
{
composing = 0;
else if (c == 0xA0)
{
- ONE_MORE_BYTE (c);
+ ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
c &= 0x7F;
}
else
}
+/* Record the starting position START and METHOD of one composition. */
+
+#define CODING_ADD_COMPOSITION_START(coding, start, method) \
+ do { \
+ struct composition_data *cmp_data = coding->cmp_data; \
+ int *data = cmp_data->data + cmp_data->used; \
+ coding->cmp_data_start = cmp_data->used; \
+ data[0] = -1; \
+ data[1] = cmp_data->char_offset + start; \
+ data[3] = (int) method; \
+ cmp_data->used += 4; \
+ } while (0)
+
+/* Record the ending position END of the current composition. */
+
+#define CODING_ADD_COMPOSITION_END(coding, end) \
+ do { \
+ struct composition_data *cmp_data = coding->cmp_data; \
+ int *data = cmp_data->data + coding->cmp_data_start; \
+ data[0] = cmp_data->used - coding->cmp_data_start; \
+ data[2] = cmp_data->char_offset + end; \
+ } while (0)
+
+/* Record one COMPONENT (alternate character or composition rule). */
+
+#define CODING_ADD_COMPOSITION_COMPONENT(coding, component) \
+ (coding->cmp_data->data[coding->cmp_data->used++] = component)
+
+
+/* Get one byte from a data pointed by SRC and increment SRC. If SRC
+ is not less than SRC_END, return -1 without incrementing Src. */
+
+#define SAFE_ONE_MORE_BYTE() (src >= src_end ? -1 : *src++)
+
+
+/* Decode a character represented as a component of composition
+ sequence of Emacs 20 style at SRC. Set C to that character, store
+ its multibyte form sequence at P, and set P to the end of that
+ sequence. If no valid character is found, set C to -1. */
+
+#define DECODE_EMACS_MULE_COMPOSITION_CHAR(c, p) \
+ do { \
+ int bytes; \
+ \
+ c = SAFE_ONE_MORE_BYTE (); \
+ if (c < 0) \
+ break; \
+ if (CHAR_HEAD_P (c)) \
+ c = -1; \
+ else if (c == 0xA0) \
+ { \
+ c = SAFE_ONE_MORE_BYTE (); \
+ if (c < 0xA0) \
+ c = -1; \
+ else \
+ { \
+ c -= 0xA0; \
+ *p++ = c; \
+ } \
+ } \
+ else if (BASE_LEADING_CODE_P (c - 0x20)) \
+ { \
+ unsigned char *p0 = p; \
+ \
+ c -= 0x20; \
+ *p++ = c; \
+ bytes = BYTES_BY_CHAR_HEAD (c); \
+ while (--bytes) \
+ { \
+ c = SAFE_ONE_MORE_BYTE (); \
+ if (c < 0) \
+ break; \
+ *p++ = c; \
+ } \
+ if (UNIBYTE_STR_AS_MULTIBYTE_P (p0, p - p0, bytes)) \
+ c = STRING_CHAR (p0, bytes); \
+ else \
+ c = -1; \
+ } \
+ else \
+ c = -1; \
+ } while (0)
+
+
+/* Decode a composition rule represented as a component of composition
+ sequence of Emacs 20 style at SRC. Set C to the rule. If not
+ valid rule is found, set C to -1. */
+
+#define DECODE_EMACS_MULE_COMPOSITION_RULE(c) \
+ do { \
+ c = SAFE_ONE_MORE_BYTE (); \
+ c -= 0xA0; \
+ if (c < 0 || c >= 81) \
+ c = -1; \
+ else \
+ { \
+ gref = c / 9, nref = c % 9; \
+ c = COMPOSITION_ENCODE_RULE (gref, nref); \
+ } \
+ } while (0)
+
+
+/* Decode composition sequence encoded by `emacs-mule' at the source
+ pointed by SRC. SRC_END is the end of source. Store information
+ of the composition in CODING->cmp_data.
+
+ For backward compatibility, decode also a composition sequence of
+ Emacs 20 style. In that case, the composition sequence contains
+ characters that should be extracted into a buffer or string. Store
+ those characters at *DESTINATION in multibyte form.
+
+ If we encounter an invalid byte sequence, return 0.
+ If we encounter an insufficient source or destination, or
+ insufficient space in CODING->cmp_data, return 1.
+ Otherwise, return consumed bytes in the source.
+
+*/
+static INLINE int
+decode_composition_emacs_mule (coding, src, src_end,
+ destination, dst_end, dst_bytes)
+ struct coding_system *coding;
+ unsigned char *src, *src_end, **destination, *dst_end;
+ int dst_bytes;
+{
+ unsigned char *dst = *destination;
+ int method, data_len, nchars;
+ unsigned char *src_base = src++;
+ /* Store components of composition. */
+ int component[COMPOSITION_DATA_MAX_BUNCH_LENGTH];
+ int ncomponent;
+ /* Store multibyte form of characters to be composed. This is for
+ Emacs 20 style composition sequence. */
+ unsigned char buf[MAX_COMPOSITION_COMPONENTS * MAX_MULTIBYTE_LENGTH];
+ unsigned char *bufp = buf;
+ int c, i, gref, nref;
+
+ if (coding->cmp_data->used + COMPOSITION_DATA_MAX_BUNCH_LENGTH
+ >= COMPOSITION_DATA_SIZE)
+ {
+ coding->result = CODING_FINISH_INSUFFICIENT_CMP;
+ return -1;
+ }
+
+ ONE_MORE_BYTE (c);
+ if (c - 0xF0 >= COMPOSITION_RELATIVE
+ && c - 0xF0 <= COMPOSITION_WITH_RULE_ALTCHARS)
+ {
+ int with_rule;
+
+ method = c - 0xF0;
+ with_rule = (method == COMPOSITION_WITH_RULE
+ || method == COMPOSITION_WITH_RULE_ALTCHARS);
+ ONE_MORE_BYTE (c);
+ data_len = c - 0xA0;
+ if (data_len < 4
+ || src_base + data_len > src_end)
+ return 0;
+ ONE_MORE_BYTE (c);
+ nchars = c - 0xA0;
+ if (c < 1)
+ return 0;
+ for (ncomponent = 0; src < src_base + data_len; ncomponent++)
+ {
+ if (ncomponent % 2 && with_rule)
+ {
+ ONE_MORE_BYTE (gref);
+ gref -= 32;
+ ONE_MORE_BYTE (nref);
+ nref -= 32;
+ c = COMPOSITION_ENCODE_RULE (gref, nref);
+ }
+ else
+ {
+ int bytes;
+ if (UNIBYTE_STR_AS_MULTIBYTE_P (src, src_end - src, bytes))
+ c = STRING_CHAR (src, bytes);
+ else
+ c = *src, bytes = 1;
+ src += bytes;
+ }
+ component[ncomponent] = c;
+ }
+ }
+ else
+ {
+ /* This may be an old Emacs 20 style format. See the comment at
+ the section 2 of this file. */
+ while (src < src_end && !CHAR_HEAD_P (*src)) src++;
+ if (src == src_end
+ && !(coding->mode & CODING_MODE_LAST_BLOCK))
+ goto label_end_of_loop;
+
+ src_end = src;
+ src = src_base + 1;
+ if (c < 0xC0)
+ {
+ method = COMPOSITION_RELATIVE;
+ for (ncomponent = 0; ncomponent < MAX_COMPOSITION_COMPONENTS;)
+ {
+ DECODE_EMACS_MULE_COMPOSITION_CHAR (c, bufp);
+ if (c < 0)
+ break;
+ component[ncomponent++] = c;
+ }
+ if (ncomponent < 2)
+ return 0;
+ nchars = ncomponent;
+ }
+ else if (c == 0xFF)
+ {
+ method = COMPOSITION_WITH_RULE;
+ src++;
+ DECODE_EMACS_MULE_COMPOSITION_CHAR (c, bufp);
+ if (c < 0)
+ return 0;
+ component[0] = c;
+ for (ncomponent = 1;
+ ncomponent < MAX_COMPOSITION_COMPONENTS * 2 - 1;)
+ {
+ DECODE_EMACS_MULE_COMPOSITION_RULE (c);
+ if (c < 0)
+ break;
+ component[ncomponent++] = c;
+ DECODE_EMACS_MULE_COMPOSITION_CHAR (c, bufp);
+ if (c < 0)
+ break;
+ component[ncomponent++] = c;
+ }
+ if (ncomponent < 3)
+ return 0;
+ nchars = (ncomponent + 1) / 2;
+ }
+ else
+ return 0;
+ }
+
+ if (buf == bufp || dst + (bufp - buf) <= (dst_bytes ? dst_end : src))
+ {
+ CODING_ADD_COMPOSITION_START (coding, coding->produced_char, method);
+ for (i = 0; i < ncomponent; i++)
+ CODING_ADD_COMPOSITION_COMPONENT (coding, component[i]);
+ CODING_ADD_COMPOSITION_END (coding, coding->produced_char + nchars);
+ if (buf < bufp)
+ {
+ unsigned char *p = buf;
+ EMIT_BYTES (p, bufp);
+ *destination += bufp - buf;
+ coding->produced_char += nchars;
+ }
+ return (src - src_base);
+ }
+ label_end_of_loop:
+ return -1;
+}
+
/* See the above "GENERAL NOTES on `decode_coding_XXX ()' functions". */
static void
if (*src == '\r')
{
- int c;
+ int c = *src++;
- src++;
if (coding->eol_type == CODING_EOL_CR)
c = '\n';
else if (coding->eol_type == CODING_EOL_CRLF)
coding->produced_char++;
continue;
}
+ else if (*src == 0x80)
+ {
+ /* Start of composition data. */
+ int consumed = decode_composition_emacs_mule (coding, src, src_end,
+ &dst, dst_end,
+ dst_bytes);
+ if (consumed < 0)
+ goto label_end_of_loop;
+ else if (consumed > 0)
+ {
+ src += consumed;
+ continue;
+ }
+ bytes = CHAR_STRING (*src, tmp);
+ p = tmp;
+ src++;
+ }
else if (UNIBYTE_STR_AS_MULTIBYTE_P (src, src_end - src, bytes))
{
p = src;
coding->produced = dst - destination;
}
-#define encode_coding_emacs_mule(coding, source, destination, src_bytes, dst_bytes) \
- encode_eol (coding, source, destination, src_bytes, dst_bytes)
+/* Encode composition data stored at DATA into a special byte sequence
+ starting by 0x80. Update CODING->cmp_data_start and maybe
+ CODING->cmp_data for the next call. */
+
+#define ENCODE_COMPOSITION_EMACS_MULE(coding, data) \
+ do { \
+ unsigned char buf[1024], *p0 = buf, *p; \
+ int len = data[0]; \
+ int i; \
+ \
+ buf[0] = 0x80; \
+ buf[1] = 0xF0 + data[3]; /* METHOD */ \
+ buf[3] = 0xA0 + (data[2] - data[1]); /* COMPOSED-CHARS */ \
+ p = buf + 4; \
+ if (data[3] == COMPOSITION_WITH_RULE \
+ || data[3] == COMPOSITION_WITH_RULE_ALTCHARS) \
+ { \
+ p += CHAR_STRING (data[4], p); \
+ for (i = 5; i < len; i += 2) \
+ { \
+ int gref, nref; \
+ COMPOSITION_DECODE_RULE (data[i], gref, nref); \
+ *p++ = 0x20 + gref; \
+ *p++ = 0x20 + nref; \
+ p += CHAR_STRING (data[i + 1], p); \
+ } \
+ } \
+ else \
+ { \
+ for (i = 4; i < len; i++) \
+ p += CHAR_STRING (data[i], p); \
+ } \
+ buf[2] = 0xA0 + (p - buf); /* COMPONENTS-BYTES */ \
+ \
+ if (dst + (p - buf) + 4 > (dst_bytes ? dst_end : src)) \
+ { \
+ coding->result = CODING_FINISH_INSUFFICIENT_DST; \
+ goto label_end_of_loop; \
+ } \
+ while (p0 < p) \
+ *dst++ = *p0++; \
+ coding->cmp_data_start += data[0]; \
+ if (coding->cmp_data_start == coding->cmp_data->used \
+ && coding->cmp_data->next) \
+ { \
+ coding->cmp_data = coding->cmp_data->next; \
+ coding->cmp_data_start = 0; \
+ } \
+ } while (0)
+
+
+static void encode_eol P_ ((struct coding_system *, unsigned char *,
+ unsigned char *, int, int));
+
+static void
+encode_coding_emacs_mule (coding, source, destination, src_bytes, dst_bytes)
+ struct coding_system *coding;
+ unsigned char *source, *destination;
+ int src_bytes, dst_bytes;
+{
+ unsigned char *src = source;
+ unsigned char *src_end = source + src_bytes;
+ unsigned char *dst = destination;
+ unsigned char *dst_end = destination + dst_bytes;
+ unsigned char *src_base;
+ int c;
+ int char_offset;
+ int *data;
+
+ Lisp_Object translation_table;
+
+ translation_table = Qnil;
+
+ /* Optimization for the case that there's no composition. */
+ if (!coding->cmp_data || coding->cmp_data->used == 0)
+ {
+ encode_eol (coding, source, destination, src_bytes, dst_bytes);
+ return;
+ }
+
+ char_offset = coding->cmp_data->char_offset;
+ data = coding->cmp_data->data + coding->cmp_data_start;
+ while (1)
+ {
+ src_base = src;
+
+ /* If SRC starts a composition, encode the information about the
+ composition in advance. */
+ if (coding->cmp_data_start < coding->cmp_data->used
+ && char_offset + coding->consumed_char == data[1])
+ {
+ ENCODE_COMPOSITION_EMACS_MULE (coding, data);
+ char_offset = coding->cmp_data->char_offset;
+ data = coding->cmp_data->data + coding->cmp_data_start;
+ }
+
+ ONE_MORE_CHAR (c);
+ if (c == '\n' && (coding->eol_type == CODING_EOL_CRLF
+ || coding->eol_type == CODING_EOL_CR))
+ {
+ if (coding->eol_type == CODING_EOL_CRLF)
+ EMIT_TWO_BYTES ('\r', c);
+ else
+ EMIT_ONE_BYTE ('\r');
+ }
+ else if (SINGLE_BYTE_CHAR_P (c))
+ EMIT_ONE_BYTE (c);
+ else
+ EMIT_BYTES (src_base, src);
+ coding->consumed_char++;
+ }
+ label_end_of_loop:
+ coding->consumed = src_base - source;
+ coding->produced = coding->produced_char = dst - destination;
+ return;
+}
\f
/*** 3. ISO2022 handlers ***/
/* The following note describes the coding system ISO2022 briefly.
Since the intention of this note is to help understand the
- functions in this file, some parts are NOT ACCURATE or OVERLY
+ functions in this file, some parts are NOT ACCURATE or are OVERLY
SIMPLIFIED. For thorough understanding, please refer to the
- original document of ISO2022.
+ original document of ISO2022. This is equivalent to the standard
+ ECMA-35, obtainable from <URL:http://www.ecma.ch/> (*).
ISO2022 provides many mechanisms to encode several character sets
- in 7-bit and 8-bit environments. For 7-bite environments, all text
+ in 7-bit and 8-bit environments. For 7-bit environments, all text
is encoded using bytes less than 128. This may make the encoded
text a little bit longer, but the text passes more easily through
- several gateways, some of which strip off MSB (Most Signigant Bit).
+ several types of gateway, some of which strip off the MSB (Most
+ Significant Bit).
- There are two kinds of character sets: control character set and
- graphic character set. The former contains control characters such
+ There are two kinds of character sets: control character sets and
+ graphic character sets. The former contain control characters such
as `newline' and `escape' to provide control functions (control
functions are also provided by escape sequences). The latter
- contains graphic characters such as 'A' and '-'. Emacs recognizes
+ contain graphic characters such as 'A' and '-'. Emacs recognizes
two control character sets and many graphic character sets.
Graphic character sets are classified into one of the following
- DIMENSION2_CHARS96
In addition, each character set is assigned an identification tag,
- unique for each set, called "final character" (denoted as <F>
+ unique for each set, called the "final character" (denoted as <F>
hereafter). The <F> of each character set is decided by ECMA(*)
when it is registered in ISO. The code range of <F> is 0x30..0x7F
(0x30..0x3F are for private use only).
Note (*): ECMA = European Computer Manufacturers Association
- Here are examples of graphic character set [NAME(<F>)]:
+ Here are examples of graphic character sets [NAME(<F>)]:
o DIMENSION1_CHARS94 -- ASCII('B'), right-half-of-JISX0201('I'), ...
o DIMENSION1_CHARS96 -- right-half-of-ISO8859-1('A'), ...
o DIMENSION2_CHARS94 -- GB2312('A'), JISX0208('B'), ...
Note (**): If <F> is '@', 'A', or 'B', the intermediate character
'(' can be omitted. We refer to this as "short-form" hereafter.
- Now you may notice that there are a lot of ways for encoding the
+ Now you may notice that there are a lot of ways of encoding the
same multilingual text in ISO2022. Actually, there exist many
coding systems such as Compound Text (used in X11's inter client
- communication, ISO-2022-JP (used in Japanese internet), ISO-2022-KR
- (used in Korean internet), EUC (Extended UNIX Code, used in Asian
+ communication, ISO-2022-JP (used in Japanese Internet), ISO-2022-KR
+ (used in Korean Internet), EUC (Extended UNIX Code, used in Asian
localized platforms), and all of these are variants of ISO2022.
In addition to the above, Emacs handles two more kinds of escape
o ESC '3' -- start relative composition with alternate chars (**)
o ESC '4' -- start rule-base composition with alternate chars (**)
Since these are not standard escape sequences of any ISO standard,
- the use of them for these meaning is restricted to Emacs only.
+ the use of them with these meanings is restricted to Emacs only.
- (*) This form is used only in Emacs 20.5 and the older versions,
+ (*) This form is used only in Emacs 20.5 and older versions,
but the newer versions can safely decode it.
- (**) This form is used only in Emacs 21.1 and the newer versions,
+ (**) This form is used only in Emacs 21.1 and newer versions,
and the older versions can't decode it.
- Here's a list of examples usages of these composition escape
+ Here's a list of example usages of these composition escape
sequences (categorized by `enum composition_method').
COMPOSITION_RELATIVE:
ESC 0 CHAR [ CHAR ] ESC 1
- COMPOSITOIN_WITH_RULE:
+ COMPOSITION_WITH_RULE:
ESC 2 CHAR [ RULE CHAR ] ESC 1
COMPOSITION_WITH_ALTCHARS:
ESC 3 ALTCHAR [ ALTCHAR ] ESC 0 CHAR [ CHAR ] ESC 1
(CODING_SPEC_ISO_INITIAL_DESIGNATION (coding_system_table[idx], 1) >= 0)
/* See the above "GENERAL NOTES on `detect_coding_XXX ()' functions".
- Check if a text is encoded in ISO2022. If it is, returns an
+ Check if a text is encoded in ISO2022. If it is, return an
integer in which appropriate flag bits any of:
CODING_CATEGORY_MASK_ISO_7
CODING_CATEGORY_MASK_ISO_7_TIGHT
are set. If a code which should never appear in ISO2022 is found,
returns 0. */
-int
-detect_coding_iso2022 (src, src_end)
+static int
+detect_coding_iso2022 (src, src_end, multibytep)
unsigned char *src, *src_end;
+ int multibytep;
{
int mask = CODING_CATEGORY_MASK_ISO;
int mask_found = 0;
int reg[4], shift_out = 0, single_shifting = 0;
- int c, c1, i, charset;
+ int c, c1, charset;
/* Dummy for ONE_MORE_BYTE. */
struct coding_system dummy_coding;
struct coding_system *coding = &dummy_coding;
reg[0] = CHARSET_ASCII, reg[1] = reg[2] = reg[3] = -1;
while (mask && src < src_end)
{
- ONE_MORE_BYTE (c);
+ ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
switch (c)
{
case ISO_CODE_ESC:
if (inhibit_iso_escape_detection)
break;
single_shifting = 0;
- ONE_MORE_BYTE (c);
+ ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
if (c >= '(' && c <= '/')
{
/* Designation sequence for a charset of dimension 1. */
- ONE_MORE_BYTE (c1);
+ ONE_MORE_BYTE_CHECK_MULTIBYTE (c1, multibytep);
if (c1 < ' ' || c1 >= 0x80
|| (charset = iso_charset_table[0][c >= ','][c1]) < 0)
/* Invalid designation sequence. Just ignore. */
else if (c == '$')
{
/* Designation sequence for a charset of dimension 2. */
- ONE_MORE_BYTE (c);
+ ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
if (c >= '@' && c <= 'B')
/* Designation for JISX0208.1978, GB2312, or JISX0208. */
reg[0] = charset = iso_charset_table[1][0][c];
else if (c >= '(' && c <= '/')
{
- ONE_MORE_BYTE (c1);
+ ONE_MORE_BYTE_CHECK_MULTIBYTE (c1, multibytep);
if (c1 < ' ' || c1 >= 0x80
|| (charset = iso_charset_table[1][c >= ','][c1]) < 0)
/* Invalid designation sequence. Just ignore. */
int i = 1;
while (src < src_end)
{
- ONE_MORE_BYTE (c);
+ ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
if (c < 0xA0)
break;
i++;
coding->cmp_data_start = 0;
}
-/* Record the starting position START and METHOD of one composition. */
-
-#define CODING_ADD_COMPOSITION_START(coding, start, method) \
- do { \
- struct composition_data *cmp_data = coding->cmp_data; \
- int *data = cmp_data->data + cmp_data->used; \
- coding->cmp_data_start = cmp_data->used; \
- data[0] = -1; \
- data[1] = cmp_data->char_offset + start; \
- data[3] = (int) method; \
- cmp_data->used += 4; \
- } while (0)
-
-/* Record the ending position END of the current composition. */
-
-#define CODING_ADD_COMPOSITION_END(coding, end) \
- do { \
- struct composition_data *cmp_data = coding->cmp_data; \
- int *data = cmp_data->data + coding->cmp_data_start; \
- data[0] = cmp_data->used - coding->cmp_data_start; \
- data[2] = cmp_data->char_offset + end; \
- } while (0)
-
-/* Record one COMPONENT (alternate character or composition rule). */
-
-#define CODING_ADD_COMPOSITION_COMPONENT(coding, component) \
- (coding->cmp_data->data[coding->cmp_data->used++] = component)
-
-/* Handle compositoin start sequence ESC 0, ESC 2, ESC 3, or ESC 4. */
+/* Handle composition start sequence ESC 0, ESC 2, ESC 3, or ESC 4.
+ ESC 0 : relative composition : ESC 0 CHAR ... ESC 1
+ ESC 2 : rulebase composition : ESC 2 CHAR RULE CHAR RULE ... CHAR ESC 1
+ ESC 3 : altchar composition : ESC 3 ALT ... ESC 0 CHAR ... ESC 1
+ ESC 4 : alt&rule composition : ESC 4 ALT RULE .. ALT ESC 0 CHAR ... ESC 1
+ */
#define DECODE_COMPOSITION_START(c1) \
do { \
that coding->cmp_data has enough space to store the \
information about the composition. If not, terminate the \
current decoding loop, allocate one more memory block for \
- coding->cmp_data in the calller, then start the decoding \
+ coding->cmp_data in the caller, then start the decoding \
loop again. We can't allocate memory here directly because \
it may cause buffer/string relocation. */ \
if (!coding->cmp_data \
} \
} while (0)
-/* Handle compositoin end sequence ESC 1. */
+/* Handle composition end sequence ESC 1. */
#define DECODE_COMPOSITION_END(c1) \
do { \
goto label_invalid_code;
/* For the moment, nested direction is not supported.
So, `coding->mode & CODING_MODE_DIRECTION' zero means
- left-to-right, and nozero means right-to-left. */
+ left-to-right, and nonzero means right-to-left. */
ONE_MORE_BYTE (c1);
switch (c1)
{
/*
It is not enough to say just "ISO2022" on encoding, we have to
- specify more details. In Emacs, each coding system of ISO2022
+ specify more details. In Emacs, each ISO2022 coding system
variant has the following specifications:
- 1. Initial designation to G0 thru G3.
+ 1. Initial designation to G0 through G3.
2. Allows short-form designation?
3. ASCII should be designated to G0 before control characters?
4. ASCII should be designated to G0 at end of line?
/* COMPOSITION_WITH_ALTCHARS or COMPOSITION_WITH_RULE_ALTCHAR */
if (coding->cmp_data_index == coding->cmp_data_start + data[0])
/* We have consumed components of the composition.
- What follows in SRC is the compositions's base
+ What follows in SRC is the composition's base
text. */
ENCODE_COMPOSITION_FAKE_START (coding);
else
\f
/*** 4. SJIS and BIG5 handlers ***/
-/* Although SJIS and BIG5 are not ISO's coding system, they are used
+/* Although SJIS and BIG5 are not ISO coding systems, they are used
quite widely. So, for the moment, Emacs supports them in the bare
C code. But, in the future, they may be supported only by CCL. */
as is. A character of charset katakana-jisx0201 is encoded by
"position-code + 0x80". A character of charset japanese-jisx0208
is encoded in 2-byte but two position-codes are divided and shifted
- so that it fit in the range below.
+ so that it fits in the range below.
--- CODE RANGE of SJIS ---
(character set) (range)
ASCII 0x00 .. 0x7F
- KATAKANA-JISX0201 0xA0 .. 0xDF
+ KATAKANA-JISX0201 0xA1 .. 0xDF
JISX0208 (1st byte) 0x81 .. 0x9F and 0xE0 .. 0xEF
(2nd byte) 0x40 .. 0x7E and 0x80 .. 0xFC
-------------------------------
/* BIG5 is a coding system encoding two character sets: ASCII and
Big5. An ASCII character is encoded as is. Big5 is a two-byte
- character set and is encoded in two-byte.
+ character set and is encoded in two bytes.
--- CODE RANGE of BIG5 ---
(character set) (range)
Check if a text is encoded in SJIS. If it is, return
CODING_CATEGORY_MASK_SJIS, else return 0. */
-int
-detect_coding_sjis (src, src_end)
+static int
+detect_coding_sjis (src, src_end, multibytep)
unsigned char *src, *src_end;
+ int multibytep;
{
int c;
/* Dummy for ONE_MORE_BYTE. */
while (1)
{
- ONE_MORE_BYTE (c);
- if (c >= 0x81)
+ ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
+ if (c < 0x80)
+ continue;
+ if (c == 0x80 || c == 0xA0 || c > 0xEF)
+ return 0;
+ if (c <= 0x9F || c >= 0xE0)
{
- if (c <= 0x9F || (c >= 0xE0 && c <= 0xEF))
- {
- ONE_MORE_BYTE (c);
- if (c < 0x40 || c == 0x7F || c > 0xFC)
- return 0;
- }
- else if (c > 0xDF)
+ ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
+ if (c < 0x40 || c == 0x7F || c > 0xFC)
return 0;
}
}
Check if a text is encoded in BIG5. If it is, return
CODING_CATEGORY_MASK_BIG5, else return 0. */
-int
-detect_coding_big5 (src, src_end)
+static int
+detect_coding_big5 (src, src_end, multibytep)
unsigned char *src, *src_end;
+ int multibytep;
{
int c;
/* Dummy for ONE_MORE_BYTE. */
while (1)
{
- ONE_MORE_BYTE (c);
- if (c >= 0xA1)
- {
- ONE_MORE_BYTE (c);
- if (c < 0x40 || (c >= 0x7F && c <= 0xA0))
- return 0;
- }
+ ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
+ if (c < 0x80)
+ continue;
+ if (c < 0xA1 || c > 0xFE)
+ return 0;
+ ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
+ if (c < 0x40 || (c > 0x7F && c < 0xA1) || c > 0xFE)
+ return 0;
}
label_end_of_loop:
return CODING_CATEGORY_MASK_BIG5;
#define UTF_8_5_OCTET_LEADING_P(c) (((c) & 0xFC) == 0xF8)
#define UTF_8_6_OCTET_LEADING_P(c) (((c) & 0xFE) == 0xFC)
-int
-detect_coding_utf_8 (src, src_end)
+static int
+detect_coding_utf_8 (src, src_end, multibytep)
unsigned char *src, *src_end;
+ int multibytep;
{
unsigned char c;
int seq_maybe_bytes;
while (1)
{
- ONE_MORE_BYTE (c);
+ ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
if (UTF_8_1_OCTET_P (c))
continue;
else if (UTF_8_2_OCTET_LEADING_P (c))
do
{
- ONE_MORE_BYTE (c);
+ ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
if (!UTF_8_EXTRA_OCTET_P (c))
return 0;
seq_maybe_bytes--;
#define UTF_16_LOW_SURROGATE_P(val) \
(((val) & 0xDC00) == 0xDC00)
-int
-detect_coding_utf_16 (src, src_end)
+static int
+detect_coding_utf_16 (src, src_end, multibytep)
unsigned char *src, *src_end;
+ int multibytep;
{
unsigned char c1, c2;
/* Dummy for TWO_MORE_BYTES. */
struct coding_system dummy_coding;
struct coding_system *coding = &dummy_coding;
- TWO_MORE_BYTES (c1, c2);
+ ONE_MORE_BYTE_CHECK_MULTIBYTE (c1, multibytep);
+ ONE_MORE_BYTE_CHECK_MULTIBYTE (c2, multibytep);
if ((c1 == 0xFF) && (c2 == 0xFE))
return CODING_CATEGORY_MASK_UTF_16_LE;
{
if (sjis_p)
{
- if (c1 >= 0xF0)
+ if (c1 == 0x80 || c1 == 0xA0 || c1 > 0xEF)
goto label_invalid_code;
- if (c1 < 0xA0 || c1 >= 0xE0)
+ if (c1 <= 0x9F || c1 >= 0xE0)
{
/* SJIS -> JISX0208 */
ONE_MORE_BYTE (c2);
else
{
/* BIG5 -> Big5 */
- if (c1 < 0xA1 || c1 > 0xFE)
+ if (c1 < 0xA0 || c1 > 0xFE)
goto label_invalid_code;
ONE_MORE_BYTE (c2);
if (c2 < 0x40 || (c2 > 0x7E && c2 < 0xA1) || c2 > 0xFE)
encoder/decoder are written in CCL program. If it is, return
CODING_CATEGORY_MASK_CCL, else return 0. */
-int
-detect_coding_ccl (src, src_end)
+static int
+detect_coding_ccl (src, src_end, multibytep)
unsigned char *src, *src_end;
+ int multibytep;
{
unsigned char *valid;
int c;
valid = coding_system_table[CODING_CATEGORY_IDX_CCL]->spec.ccl.valid_codes;
while (1)
{
- ONE_MORE_BYTE (c);
+ ONE_MORE_BYTE_CHECK_MULTIBYTE (c, multibytep);
if (! valid[c])
return 0;
}
/* See "GENERAL NOTES about `encode_coding_XXX ()' functions". Encode
format of end-of-line according to `coding->eol_type'. It also
- convert multibyte form 8-bit characers to unibyte if
+ convert multibyte form 8-bit characters to unibyte if
CODING->src_multibyte is nonzero. If `coding->mode &
CODING_MODE_SELECTIVE_DISPLAY' is nonzero, code '\r' in source text
also means end-of-line. */
\f
/*** 7. C library functions ***/
-/* In Emacs Lisp, coding system is represented by a Lisp symbol which
+/* In Emacs Lisp, a coding system is represented by a Lisp symbol which
has a property `coding-system'. The value of this property is a
- vector of length 5 (called as coding-vector). Among elements of
+ vector of length 5 (called the coding-vector). Among elements of
this vector, the first (element[0]) and the fifth (element[4])
carry important information for decoding/encoding. Before
decoding/encoding, this information should be set in fields of a
structure of type `coding_system'.
- A value of property `coding-system' can be a symbol of another
+ The value of the property `coding-system' can be a symbol of another
subsidiary coding-system. In that case, Emacs gets coding-vector
from that symbol.
If `coding->type' takes the other value, element[4] is ignored.
- Emacs Lisp's coding system also carries information about format of
+ Emacs Lisp's coding systems also carry information about format of
end-of-line in a value of property `eol-type'. If the value is
integer, 0 means CODING_EOL_LF, 1 means CODING_EOL_CRLF, and 2
means CODING_EOL_CR. If it is not integer, it should be a vector
of subsidiary coding systems of which property `eol-type' has one
- of above values.
+ of the above values.
*/
{
Lisp_Object coding_spec, coding_type, eol_type, plist;
Lisp_Object val;
- int i;
+
+ /* At first, zero clear all members. */
+ bzero (coding, sizeof (struct coding_system));
/* Initialize some fields required for all kinds of coding systems. */
coding->symbol = coding_system;
- coding->common_flags = 0;
- coding->mode = 0;
coding->heading_ascii = -1;
coding->post_read_conversion = coding->pre_write_conversion = Qnil;
coding->composing = COMPOSITION_DISABLED;
}
else
coding->type = coding_type_no_conversion;
+ /* Initialize this member. Any thing other than
+ CODING_CATEGORY_IDX_UTF_16_BE and
+ CODING_CATEGORY_IDX_UTF_16_LE are ok because they have
+ special treatment in detect_eol. */
+ coding->category_idx = CODING_CATEGORY_IDX_EMACS_MULE;
+
return 0;
}
`translation-table-for-decode', `translation-table-for-encode'. */
plist = XVECTOR (coding_spec)->contents[3];
/* Pre & post conversion functions should be disabled if
- inhibit_eol_conversion is nozero. This is the case that a code
+ inhibit_eol_conversion is nonzero. This is the case that a code
conversion function is called while those functions are running. */
if (! inhibit_pre_post_conversion)
{
{
case 0:
coding->type = coding_type_emacs_mule;
+ coding->common_flags
+ |= CODING_REQUIRE_DECODING_MASK | CODING_REQUIRE_ENCODING_MASK;
+ coding->composing = COMPOSITION_NO;
if (!NILP (coding->post_read_conversion))
coding->common_flags |= CODING_REQUIRE_DECODING_MASK;
if (!NILP (coding->pre_write_conversion))
}
coding->common_flags |= CODING_REQUIRE_FLUSHING_MASK;
coding->spec.ccl.cr_carryover = 0;
+ coding->spec.ccl.eight_bit_carryover[0] = 0;
break;
case 5:
o coding-category-iso-7-else
The category for a coding system which has the same code range
- as ISO2022 of 7-bit environemnt but uses locking shift or
+ as ISO2022 of 7-bit environment but uses locking shift or
single shift functions. Assigned the coding-system (Lisp
symbol) `iso-2022-7bit-lock' by default.
o coding-category-iso-8-else
The category for a coding system which has the same code range
- as ISO2022 of 8-bit environemnt but uses locking shift or
+ as ISO2022 of 8-bit environment but uses locking shift or
single shift functions. Assigned the coding-system (Lisp
symbol) `iso-2022-8bit-ss2' by default.
`no-conversion' by default.
Each of them is a Lisp symbol and the value is an actual
- `coding-system's (this is also a Lisp symbol) assigned by a user.
+ `coding-system' (this is also a Lisp symbol) assigned by a user.
What Emacs does actually is to detect a category of coding system.
Then, it uses a `coding-system' assigned to it. If Emacs can't
- decide only one possible category, it selects a category of the
+ decide a single possible category, it selects a category of the
highest priority. Priorities of categories are also specified by a
user in a Lisp variable `coding-category-list'.
CODING_CATEGORY_MASK_XXX in `coding.h'. If PRIORITIES is non-NULL,
it should point the table `coding_priorities'. In that case, only
the flag bit for a coding system of the highest priority is set in
- the returned value.
+ the returned value. If MULTIBYTEP is nonzero, 8-bit codes of the
+ range 0x80..0x9F are in multibyte form.
How many ASCII characters are at the head is returned as *SKIP. */
static int
-detect_coding_mask (source, src_bytes, priorities, skip)
+detect_coding_mask (source, src_bytes, priorities, skip, multibytep)
unsigned char *source;
int src_bytes, *priorities, *skip;
+ int multibytep;
{
register unsigned char c;
unsigned char *src = source, *src_end = source + src_bytes;
unsigned int mask, utf16_examined_p, iso2022_examined_p;
- int i, idx;
+ int i;
/* At first, skip all ASCII characters and control characters except
for three ISO2022 specific control characters. */
{
/* i.e. (c == ISO_CODE_ESC || c == ISO_CODE_SI || c == ISO_CODE_SO) */
/* C is an ISO2022 specific control code of C0. */
- mask = detect_coding_iso2022 (src, src_end);
+ mask = detect_coding_iso2022 (src, src_end, multibytep);
if (mask == 0)
{
/* No valid ISO2022 code follows C. Try again. */
{
int try;
+ if (multibytep && c == LEADING_CODE_8_BIT_CONTROL)
+ c = src[1] - 0x20;
+
if (c < 0xA0)
{
/* C is the first byte of SJIS character code,
iso2022_examined_p = 1;
}
else if (priorities[i] & try & CODING_CATEGORY_MASK_SJIS)
- mask |= detect_coding_sjis (src, src_end);
+ mask |= detect_coding_sjis (src, src_end, multibytep);
else if (priorities[i] & try & CODING_CATEGORY_MASK_UTF_8)
- mask |= detect_coding_utf_8 (src, src_end);
+ mask |= detect_coding_utf_8 (src, src_end, multibytep);
else if (!utf16_examined_p
&& (priorities[i] & try &
CODING_CATEGORY_MASK_UTF_16_BE_LE))
{
- mask |= detect_coding_utf_16 (src, src_end);
+ mask |= detect_coding_utf_16 (src, src_end, multibytep);
utf16_examined_p = 1;
}
else if (priorities[i] & try & CODING_CATEGORY_MASK_BIG5)
- mask |= detect_coding_big5 (src, src_end);
+ mask |= detect_coding_big5 (src, src_end, multibytep);
else if (priorities[i] & try & CODING_CATEGORY_MASK_EMACS_MULE)
- mask |= detect_coding_emacs_mule (src, src_end);
+ mask |= detect_coding_emacs_mule (src, src_end, multibytep);
else if (priorities[i] & try & CODING_CATEGORY_MASK_CCL)
- mask |= detect_coding_ccl (src, src_end);
+ mask |= detect_coding_ccl (src, src_end, multibytep);
else if (priorities[i] & CODING_CATEGORY_MASK_RAW_TEXT)
mask |= CODING_CATEGORY_MASK_RAW_TEXT;
else if (priorities[i] & CODING_CATEGORY_MASK_BINARY)
return CODING_CATEGORY_MASK_RAW_TEXT;
}
if (try & CODING_CATEGORY_MASK_ISO)
- mask |= detect_coding_iso2022 (src, src_end);
+ mask |= detect_coding_iso2022 (src, src_end, multibytep);
if (try & CODING_CATEGORY_MASK_SJIS)
- mask |= detect_coding_sjis (src, src_end);
+ mask |= detect_coding_sjis (src, src_end, multibytep);
if (try & CODING_CATEGORY_MASK_BIG5)
- mask |= detect_coding_big5 (src, src_end);
+ mask |= detect_coding_big5 (src, src_end, multibytep);
if (try & CODING_CATEGORY_MASK_UTF_8)
- mask |= detect_coding_utf_8 (src, src_end);
+ mask |= detect_coding_utf_8 (src, src_end, multibytep);
if (try & CODING_CATEGORY_MASK_UTF_16_BE_LE)
- mask |= detect_coding_utf_16 (src, src_end);
+ mask |= detect_coding_utf_16 (src, src_end, multibytep);
if (try & CODING_CATEGORY_MASK_EMACS_MULE)
- mask |= detect_coding_emacs_mule (src, src_end);
+ mask |= detect_coding_emacs_mule (src, src_end, multibytep);
if (try & CODING_CATEGORY_MASK_CCL)
- mask |= detect_coding_ccl (src, src_end);
+ mask |= detect_coding_ccl (src, src_end, multibytep);
}
return (mask | CODING_CATEGORY_MASK_RAW_TEXT | CODING_CATEGORY_MASK_BINARY);
}
int src_bytes;
{
unsigned int idx;
- int skip, mask, i;
+ int skip, mask;
Lisp_Object val;
val = Vcoding_category_list;
- mask = detect_coding_mask (src, src_bytes, coding_priorities, &skip);
+ mask = detect_coding_mask (src, src_bytes, coding_priorities, &skip,
+ coding->src_multibyte);
coding->heading_ascii = skip;
if (!mask) return;
static int
detect_eol_type_in_2_octet_form (source, src_bytes, skip, big_endian_p)
unsigned char *source;
- int src_bytes, *skip;
+ int src_bytes, *skip, big_endian_p;
{
unsigned char *src = source, *src_end = src + src_bytes;
unsigned int c1, c2;
{
struct ccl_program *ccl
= encodep ? &coding->spec.ccl.encoder : &coding->spec.ccl.decoder;
- int result;
+ unsigned char *dst = destination;
+ ccl->suppress_error = coding->suppress_error;
ccl->last_block = coding->mode & CODING_MODE_LAST_BLOCK;
if (encodep)
- ccl->eol_type = coding->eol_type;
+ {
+ /* On encoding, EOL format is converted within ccl_driver. For
+ that, setup proper information in the structure CCL. */
+ ccl->eol_type = coding->eol_type;
+ if (ccl->eol_type ==CODING_EOL_UNDECIDED)
+ ccl->eol_type = CODING_EOL_LF;
+ ccl->cr_consumed = coding->spec.ccl.cr_carryover;
+ }
ccl->multibyte = coding->src_multibyte;
- coding->produced = ccl_driver (ccl, source, destination,
- src_bytes, dst_bytes, &(coding->consumed));
+ if (coding->spec.ccl.eight_bit_carryover[0] != 0)
+ {
+ /* Move carryover bytes to DESTINATION. */
+ unsigned char *p = coding->spec.ccl.eight_bit_carryover;
+ while (*p)
+ *dst++ = *p++;
+ coding->spec.ccl.eight_bit_carryover[0] = 0;
+ if (dst_bytes)
+ dst_bytes -= dst - destination;
+ }
+
+ coding->produced = (ccl_driver (ccl, source, dst, src_bytes, dst_bytes,
+ &(coding->consumed))
+ + dst - destination);
+
if (encodep)
- coding->produced_char = coding->produced;
+ {
+ coding->produced_char = coding->produced;
+ coding->spec.ccl.cr_carryover = ccl->cr_consumed;
+ }
+ else if (!ccl->eight_bit_control)
+ {
+ /* The produced bytes forms a valid multibyte sequence. */
+ coding->produced_char
+ = multibyte_chars_in_text (destination, coding->produced);
+ coding->spec.ccl.eight_bit_carryover[0] = 0;
+ }
else
{
+ /* On decoding, the destination should always multibyte. But,
+ CCL program might have been generated an invalid multibyte
+ sequence. Here we make such a sequence valid as
+ multibyte. */
int bytes
= dst_bytes ? dst_bytes : source + coding->consumed - destination;
+
+ if ((coding->consumed < src_bytes
+ || !ccl->last_block)
+ && coding->produced >= 1
+ && destination[coding->produced - 1] >= 0x80)
+ {
+ /* We should not convert the tailing 8-bit codes to
+ multibyte form even if they doesn't form a valid
+ multibyte sequence. They may form a valid sequence in
+ the next call. */
+ int carryover = 0;
+
+ if (destination[coding->produced - 1] < 0xA0)
+ carryover = 1;
+ else if (coding->produced >= 2)
+ {
+ if (destination[coding->produced - 2] >= 0x80)
+ {
+ if (destination[coding->produced - 2] < 0xA0)
+ carryover = 2;
+ else if (coding->produced >= 3
+ && destination[coding->produced - 3] >= 0x80
+ && destination[coding->produced - 3] < 0xA0)
+ carryover = 3;
+ }
+ }
+ if (carryover > 0)
+ {
+ BCOPY_SHORT (destination + coding->produced - carryover,
+ coding->spec.ccl.eight_bit_carryover,
+ carryover);
+ coding->spec.ccl.eight_bit_carryover[carryover] = 0;
+ coding->produced -= carryover;
+ }
+ }
coding->produced = str_as_multibyte (destination, bytes,
coding->produced,
&(coding->produced_char));
if (coding->eol_type == CODING_EOL_UNDECIDED
&& coding->type != coding_type_ccl)
- detect_eol (coding, source, src_bytes);
+ {
+ detect_eol (coding, source, src_bytes);
+ /* We had better recover the original eol format if we
+ encounter an inconsistent eol format while decoding. */
+ coding->mode |= CODING_MODE_INHIBIT_INCONSISTENT_EOL;
+ }
coding->produced = coding->produced_char = 0;
coding->consumed = coding->consumed_char = 0;
case CODING_CATEGORY_IDX_ISO_7:
case CODING_CATEGORY_IDX_ISO_7_TIGHT:
{
- /* We can skip all charactes at the tail except for 8-bit
+ /* We can skip all characters at the tail except for 8-bit
codes and ESC and the following 2-byte at the tail. */
unsigned char *eight_bit = NULL;
if (!NILP (CHAR_TABLE_REF (translation_table, i)))
break;
if (i < 128)
- /* Some ASCII character should be tranlsated. We give up
+ /* Some ASCII character should be translated. We give up
shrinking. */
return;
}
}
/* Reflect the saved information about compositions to OBJ.
- CODING->cmp_data points to a memory block for the informaiton. OBJ
+ CODING->cmp_data points to a memory block for the information. OBJ
is a buffer or a string, defaults to the current buffer. */
void
replace_range (insdel.c) to know what we are doing.
If REPLACE is zero, it is assumed that the source text is unibyte.
- Otherwize, it is assumed that the source text is multibyte. */
+ Otherwise, it is assumed that the source text is multibyte. */
int
code_convert_region (from, from_byte, to, to_byte, coding, encodep, replace)
int prev_Z;
int multibyte_p = !NILP (current_buffer->enable_multibyte_characters);
- coding->src_multibyte = replace && multibyte_p;
- coding->dst_multibyte = multibyte_p;
-
deletion = Qnil;
- saved_coding_symbol = Qnil;
+ saved_coding_symbol = coding->symbol;
if (from < PT && PT < to)
{
encodings again in vain. */
coding->type = coding_type_emacs_mule;
coding->category_idx = CODING_CATEGORY_IDX_EMACS_MULE;
+ /* As emacs-mule decoder will handle composition, we
+ need this setting to allocate coding->cmp_data
+ later. */
+ coding->composing = COMPOSITION_NO;
}
}
if (coding->eol_type == CODING_EOL_UNDECIDED
&& coding->type != coding_type_ccl)
{
- saved_coding_symbol = coding->symbol;
detect_eol (coding, BYTE_POS_ADDR (from_byte), len_byte);
if (coding->eol_type == CODING_EOL_UNDECIDED)
coding->eol_type = CODING_EOL_LF;
/* We had better recover the original eol format if we
- encounter an inconsitent eol format while decoding. */
+ encounter an inconsistent eol format while decoding. */
coding->mode |= CODING_MODE_INHIBIT_INCONSISTENT_EOL;
}
}
len -= total_skip; len_byte -= total_skip;
}
- /* For converion, we must put the gap before the text in addition to
+ /* For conversion, we must put the gap before the text in addition to
making the gap larger for efficient decoding. The required gap
size starts from 2000 which is the magic number used in make_gap.
But, after one batch of conversion, it will be incremented if we
if (first)
{
/* We have just done the first batch of conversion which was
- stoped because of insufficient gap. Let's reconsider the
+ stopped because of insufficient gap. Let's reconsider the
required gap size (i.e. SRT - DST) now.
We have converted ORIG bytes (== coding->consumed) into
inserted_byte = str_to_multibyte (GPT_ADDR, GAP_SIZE, inserted_byte);
}
- /* If we have shrinked the conversion area, adjust it now. */
+ /* If we shrank the conversion area, adjust it now. */
if (total_skip > 0)
{
if (tail_skip > 0)
{
int count = specpdl_ptr - specpdl;
struct gcpro gcpro1;
- struct buffer *prev = current_buffer;
int multibyte = STRING_MULTIBYTE (str);
record_unwind_protect (Fset_buffer, Fcurrent_buffer ());
{
int len;
struct conversion_buffer buf;
- int from, to, to_byte;
+ int from, to_byte;
struct gcpro gcpro1;
Lisp_Object saved_coding_symbol;
int result;
int consumed, consumed_char, produced, produced_char;
from = 0;
- to = XSTRING (str)->size;
to_byte = STRING_BYTES (XSTRING (str));
- saved_coding_symbol = Qnil;
+ saved_coding_symbol = coding->symbol;
+ coding->src_multibyte = STRING_MULTIBYTE (str);
+ coding->dst_multibyte = 1;
if (CODING_REQUIRE_DETECTION (coding))
{
/* See the comments in code_convert_region. */
{
detect_coding (coding, XSTRING (str)->data, to_byte);
if (coding->type == coding_type_undecided)
- coding->type = coding_type_emacs_mule;
+ {
+ coding->type = coding_type_emacs_mule;
+ coding->category_idx = CODING_CATEGORY_IDX_EMACS_MULE;
+ /* As emacs-mule decoder will handle composition, we
+ need this setting to allocate coding->cmp_data
+ later. */
+ coding->composing = COMPOSITION_NO;
+ }
}
if (coding->eol_type == CODING_EOL_UNDECIDED
&& coding->type != coding_type_ccl)
if (coding->eol_type == CODING_EOL_UNDECIDED)
coding->eol_type = CODING_EOL_LF;
/* We had better recover the original eol format if we
- encounter an inconsitent eol format while decoding. */
+ encounter an inconsistent eol format while decoding. */
coding->mode |= CODING_MODE_INHIBIT_INCONSISTENT_EOL;
}
}
- coding->src_multibyte = 0;
- coding->dst_multibyte = (coding->type != coding_type_no_conversion
- && coding->type != coding_type_raw_text);
+ if (coding->type == coding_type_no_conversion
+ || coding->type == coding_type_raw_text)
+ coding->dst_multibyte = 0;
+
require_decoding = CODING_REQUIRE_DECODING (coding);
if (STRING_MULTIBYTE (str))
str = Fstring_as_unibyte (str);
to_byte = STRING_BYTES (XSTRING (str));
nocopy = 1;
+ coding->src_multibyte = 0;
}
/* Try to skip the heading and tailing ASCIIs. */
extend_conversion_buffer (&buf);
else if (result == CODING_FINISH_INCONSISTENT_EOL)
{
+ Lisp_Object eol_type;
+
/* Recover the original EOL format. */
if (coding->eol_type == CODING_EOL_CR)
{
produced += num_eol;
produced_char += num_eol;
}
+ /* Suppress eol-format conversion in the further conversion. */
coding->eol_type = CODING_EOL_LF;
- coding->symbol = saved_coding_symbol;
+
+ /* Set the coding system symbol to that for Unix-like EOL. */
+ eol_type = Fget (saved_coding_symbol, Qeol_type);
+ if (VECTORP (eol_type)
+ && XVECTOR (eol_type)->size == 3
+ && SYMBOLP (XVECTOR (eol_type)->contents[CODING_EOL_LF]))
+ coding->symbol = XVECTOR (eol_type)->contents[CODING_EOL_LF];
+ else
+ coding->symbol = saved_coding_symbol;
+
+
}
}
int len;
struct conversion_buffer buf;
int from, to, to_byte;
- struct gcpro gcpro1;
- Lisp_Object saved_coding_symbol;
int result;
int shrinked_bytes = 0;
Lisp_Object newstr;
to = XSTRING (str)->size;
to_byte = STRING_BYTES (XSTRING (str));
- saved_coding_symbol = Qnil;
-
/* Encoding routines determine the multibyteness of the source text
by coding->src_multibyte. */
coding->src_multibyte = STRING_MULTIBYTE (str);
}
\f
Lisp_Object
-detect_coding_system (src, src_bytes, highest)
+detect_coding_system (src, src_bytes, highest, multibytep)
unsigned char *src;
int src_bytes, highest;
+ int multibytep;
{
int coding_mask, eol_type;
Lisp_Object val, tmp;
int dummy;
- coding_mask = detect_coding_mask (src, src_bytes, NULL, &dummy);
+ coding_mask = detect_coding_mask (src, src_bytes, NULL, &dummy, multibytep);
eol_type = detect_eol_type (src, src_bytes, &dummy);
if (eol_type == CODING_EOL_INCONSISTENT)
eol_type = CODING_EOL_UNDECIDED;
{
int from, to;
int from_byte, to_byte;
+ int include_anchor_byte = 0;
CHECK_NUMBER_COERCE_MARKER (start, 0);
CHECK_NUMBER_COERCE_MARKER (end, 1);
if (from < GPT && to >= GPT)
move_gap_both (to, to_byte);
+ /* If we an anchor byte `\0' follows the region, we include it in
+ the detecting source. Then code detectors can handle the tailing
+ byte sequence more accurately.
+ Fix me: This is not an perfect solution. It is better that we
+ add one more argument, say LAST_BLOCK, to all detect_coding_XXX.
+ */
+ if (to == Z || (to == GPT && GAP_SIZE > 0))
+ include_anchor_byte = 1;
return detect_coding_system (BYTE_POS_ADDR (from_byte),
- to_byte - from_byte,
- !NILP (highest));
+ to_byte - from_byte + include_anchor_byte,
+ !NILP (highest),
+ !NILP (current_buffer
+ ->enable_multibyte_characters));
}
DEFUN ("detect-coding-string", Fdetect_coding_string, Sdetect_coding_string,
CHECK_STRING (string, 0);
return detect_coding_system (XSTRING (string)->data,
- STRING_BYTES (XSTRING (string)),
- !NILP (highest));
+ /* "+ 1" is to include the anchor byte
+ `\0'. With this, code detectors can
+ handle the tailing bytes more
+ accurately. */
+ STRING_BYTES (XSTRING (string)) + 1,
+ !NILP (highest),
+ STRING_MULTIBYTE (string));
}
/* Return an intersection of lists L1 and L2. */
int non_ascii_p = 0;
int single_byte_char_found = 0;
unsigned char *p1, *p1end, *p2, *p2end, *p;
- Lisp_Object args[2];
if (STRINGP (start))
{
safe_codings = Fappend (2, args);
}
else
- safe_codings = Fcons (Qraw_text, Fcons (Qemacs_mule, safe_codings));
+ safe_codings = Fcons (Qraw_text,
+ Fcons (Qemacs_mule,
+ Fcons (Qno_conversion, safe_codings)));
return safe_codings;
}
int encodep;
{
struct coding_system coding;
- int from, to, len;
+ int from, to;
CHECK_NUMBER_COERCE_MARKER (start, 0);
CHECK_NUMBER_COERCE_MARKER (end, 1);
DEFUN ("decode-coding-region", Fdecode_coding_region, Sdecode_coding_region,
3, 3, "r\nzCoding system: ",
- "Decode the current region by specified coding system.\n\
+ "Decode the current region from the specified coding system.\n\
When called from a program, takes three arguments:\n\
START, END, and CODING-SYSTEM. START and END are buffer positions.\n\
This function sets `last-coding-system-used' to the precise coding system\n\
DEFUN ("encode-coding-region", Fencode_coding_region, Sencode_coding_region,
3, 3, "r\nzCoding system: ",
- "Encode the current region by specified coding system.\n\
+ "Encode the current region into the specified coding system.\n\
When called from a program, takes three arguments:\n\
START, END, and CODING-SYSTEM. START and END are buffer positions.\n\
This function sets `last-coding-system-used' to the precise coding system\n\
DEFUN ("decode-coding-string", Fdecode_coding_string, Sdecode_coding_string,
2, 3, 0,
"Decode STRING which is encoded in CODING-SYSTEM, and return the result.\n\
-Optional arg NOCOPY non-nil means it is ok to return STRING itself\n\
+Optional arg NOCOPY non-nil means it is OK to return STRING itself\n\
if the decoding operation is trivial.\n\
This function sets `last-coding-system-used' to the precise coding system\n\
used (which may be different from CODING-SYSTEM if CODING-SYSTEM is\n\
DEFUN ("encode-coding-string", Fencode_coding_string, Sencode_coding_string,
2, 3, 0,
"Encode STRING to CODING-SYSTEM, and return the result.\n\
-Optional arg NOCOPY non-nil means it is ok to return STRING itself\n\
+Optional arg NOCOPY non-nil means it is OK to return STRING itself\n\
if the encoding operation is trivial.\n\
This function sets `last-coding-system-used' to the precise coding system\n\
used (which may be different from CODING-SYSTEM if CODING-SYSTEM is\n\
setup_coding_system (Fcheck_coding_system (coding_system), &terminal_coding);
/* We had better not send unsafe characters to terminal. */
terminal_coding.flags |= CODING_FLAG_ISO_SAFE;
- /* Characer composition should be disabled. */
+ /* Character composition should be disabled. */
terminal_coding.composing = COMPOSITION_DISABLED;
+ /* Error notification should be suppressed. */
+ terminal_coding.suppress_error = 1;
terminal_coding.src_multibyte = 1;
terminal_coding.dst_multibyte = 0;
return Qnil;
CHECK_SYMBOL (coding_system, 0);
setup_coding_system (Fcheck_coding_system (coding_system),
&safe_terminal_coding);
- /* Characer composition should be disabled. */
+ /* Character composition should be disabled. */
safe_terminal_coding.composing = COMPOSITION_DISABLED;
+ /* Error notification should be suppressed. */
+ terminal_coding.suppress_error = 1;
safe_terminal_coding.src_multibyte = 1;
safe_terminal_coding.dst_multibyte = 0;
return Qnil;
{
CHECK_SYMBOL (coding_system, 0);
setup_coding_system (Fcheck_coding_system (coding_system), &keyboard_coding);
- /* Characer composition should be disabled. */
+ /* Character composition should be disabled. */
keyboard_coding.composing = COMPOSITION_DISABLED;
return Qnil;
}
operation = args[0];
if (!SYMBOLP (operation)
|| !INTEGERP (target_idx = Fget (operation, Qtarget_idx)))
- error ("Invalid first arguement");
+ error ("Invalid first argument");
if (nargs < 1 + XINT (target_idx))
error ("Too few arguments for operation: %s",
XSYMBOL (operation)->name->data);
target = args[XINT (target_idx) + 1];
if (!(STRINGP (target)
|| (EQ (operation, Qopen_network_stream) && INTEGERP (target))))
- error ("Invalid %dth argument", XINT (target_idx) + 1);
+ error ("Invalid argument %d", XINT (target_idx) + 1);
chain = ((EQ (operation, Qinsert_file_contents)
|| EQ (operation, Qwrite_region))
Vcoding_system_alist = Qnil;
DEFVAR_LISP ("coding-category-list", &Vcoding_category_list,
- "List of coding-categories (symbols) ordered by priority.");
+ "List of coding-categories (symbols) ordered by priority.\n\
+\n\
+On detecting a coding system, Emacs tries code detection algorithms\n\
+associated with each coding-category one by one in this order. When\n\
+one algorithm agrees with a byte sequence of source text, the coding\n\
+system bound to the corresponding coding-category is selected.");
{
int i;
DEFVAR_LISP ("standard-translation-table-for-encode",
&Vstandard_translation_table_for_encode,
- "Table for translationg characters while encoding.");
+ "Table for translating characters while encoding.");
Vstandard_translation_table_for_encode = Qnil;
DEFVAR_LISP ("charset-revision-table", &Vcharset_revision_alist,
"Alist of charsets vs revision numbers.\n\
While encoding, if a charset (car part of an element) is found,\n\
-designate it with the escape sequence identifing revision (cdr part of the element).");
+designate it with the escape sequence identifying revision (cdr part of the element).");
Vcharset_revision_alist = Qnil;
DEFVAR_LISP ("default-process-coding-system",
HCoop / bpt / emacs.git / blobdiff