1 /* Coding system handler (conversion, detection, and etc).
2 Copyright (C) 1995, 1997, 1998, 2002 Electrotechnical Laboratory, JAPAN.
3 Licensed to the Free Software Foundation.
4 Copyright (C) 2001 Free Software Foundation, Inc.
6 This file is part of GNU Emacs.
8 GNU Emacs is free software; you can redistribute it and/or modify
9 it under the terms of the GNU General Public License as published by
10 the Free Software Foundation; either version 2, or (at your option)
13 GNU Emacs is distributed in the hope that it will be useful,
14 but WITHOUT ANY WARRANTY; without even the implied warranty of
15 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
16 GNU General Public License for more details.
18 You should have received a copy of the GNU General Public License
19 along with GNU Emacs; see the file COPYING. If not, write to
20 the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
21 Boston, MA 02111-1307, USA. */
23 /*** TABLE OF CONTENTS ***
27 2. Emacs' internal format (emacs-mule) handlers
29 4. Shift-JIS and BIG5 handlers
31 6. End-of-line handlers
32 7. C library functions
33 8. Emacs Lisp library functions
38 /*** 0. General comments ***/
41 /*** GENERAL NOTE on CODING SYSTEMS ***
43 A coding system is an encoding mechanism for one or more character
44 sets. Here's a list of coding systems which Emacs can handle. When
45 we say "decode", it means converting some other coding system to
46 Emacs' internal format (emacs-mule), and when we say "encode",
47 it means converting the coding system emacs-mule to some other
50 0. Emacs' internal format (emacs-mule)
52 Emacs itself holds a multi-lingual character in buffers and strings
53 in a special format. Details are described in section 2.
57 The most famous coding system for multiple character sets. X's
58 Compound Text, various EUCs (Extended Unix Code), and coding
59 systems used in Internet communication such as ISO-2022-JP are
60 all variants of ISO2022. Details are described in section 3.
62 2. SJIS (or Shift-JIS or MS-Kanji-Code)
64 A coding system to encode character sets: ASCII, JISX0201, and
65 JISX0208. Widely used for PC's in Japan. Details are described in
70 A coding system to encode the character sets ASCII and Big5. Widely
71 used for Chinese (mainly in Taiwan and Hong Kong). Details are
72 described in section 4. In this file, when we write "BIG5"
73 (all uppercase), we mean the coding system, and when we write
74 "Big5" (capitalized), we mean the character set.
78 A coding system for text containing random 8-bit code. Emacs does
79 no code conversion on such text except for end-of-line format.
83 If a user wants to read/write text encoded in a coding system not
84 listed above, he can supply a decoder and an encoder for it as CCL
85 (Code Conversion Language) programs. Emacs executes the CCL program
86 while reading/writing.
88 Emacs represents a coding system by a Lisp symbol that has a property
89 `coding-system'. But, before actually using the coding system, the
90 information about it is set in a structure of type `struct
91 coding_system' for rapid processing. See section 6 for more details.
95 /*** GENERAL NOTES on END-OF-LINE FORMAT ***
97 How end-of-line of text is encoded depends on the operating system.
98 For instance, Unix's format is just one byte of `line-feed' code,
99 whereas DOS's format is two-byte sequence of `carriage-return' and
100 `line-feed' codes. MacOS's format is usually one byte of
103 Since text character encoding and end-of-line encoding are
104 independent, any coding system described above can have any
105 end-of-line format. So Emacs has information about end-of-line
106 format in each coding-system. See section 6 for more details.
110 /*** GENERAL NOTES on `detect_coding_XXX ()' functions ***
112 These functions check if a text between SRC and SRC_END is encoded
113 in the coding system category XXX. Each returns an integer value in
114 which appropriate flag bits for the category XXX are set. The flag
115 bits are defined in macros CODING_CATEGORY_MASK_XXX. Below is the
116 template for these functions. If MULTIBYTEP is nonzero, 8-bit codes
117 of the range 0x80..0x9F are in multibyte form. */
120 detect_coding_emacs_mule (src
, src_end
, multibytep
)
121 unsigned char *src
, *src_end
;
128 /*** GENERAL NOTES on `decode_coding_XXX ()' functions ***
130 These functions decode SRC_BYTES length of unibyte text at SOURCE
131 encoded in CODING to Emacs' internal format. The resulting
132 multibyte text goes to a place pointed to by DESTINATION, the length
133 of which should not exceed DST_BYTES.
135 These functions set the information about original and decoded texts
136 in the members `produced', `produced_char', `consumed', and
137 `consumed_char' of the structure *CODING. They also set the member
138 `result' to one of CODING_FINISH_XXX indicating how the decoding
141 DST_BYTES zero means that the source area and destination area are
142 overlapped, which means that we can produce a decoded text until it
143 reaches the head of the not-yet-decoded source text.
145 Below is a template for these functions. */
148 decode_coding_XXX (coding
, source
, destination
, src_bytes
, dst_bytes
)
149 struct coding_system
*coding
;
150 unsigned char *source
, *destination
;
151 int src_bytes
, dst_bytes
;
157 /*** GENERAL NOTES on `encode_coding_XXX ()' functions ***
159 These functions encode SRC_BYTES length text at SOURCE from Emacs'
160 internal multibyte format to CODING. The resulting unibyte text
161 goes to a place pointed to by DESTINATION, the length of which
162 should not exceed DST_BYTES.
164 These functions set the information about original and encoded texts
165 in the members `produced', `produced_char', `consumed', and
166 `consumed_char' of the structure *CODING. They also set the member
167 `result' to one of CODING_FINISH_XXX indicating how the encoding
170 DST_BYTES zero means that the source area and destination area are
171 overlapped, which means that we can produce encoded text until it
172 reaches at the head of the not-yet-encoded source text.
174 Below is a template for these functions. */
177 encode_coding_XXX (coding
, source
, destination
, src_bytes
, dst_bytes
)
178 struct coding_system
*coding
;
179 unsigned char *source
, *destination
;
180 int src_bytes
, dst_bytes
;
186 /*** COMMONLY USED MACROS ***/
188 /* The following two macros ONE_MORE_BYTE and TWO_MORE_BYTES safely
189 get one, two, and three bytes from the source text respectively.
190 If there are not enough bytes in the source, they jump to
191 `label_end_of_loop'. The caller should set variables `coding',
192 `src' and `src_end' to appropriate pointer in advance. These
193 macros are called from decoding routines `decode_coding_XXX', thus
194 it is assumed that the source text is unibyte. */
196 #define ONE_MORE_BYTE(c1) \
198 if (src >= src_end) \
200 coding->result = CODING_FINISH_INSUFFICIENT_SRC; \
201 goto label_end_of_loop; \
206 #define TWO_MORE_BYTES(c1, c2) \
208 if (src + 1 >= src_end) \
210 coding->result = CODING_FINISH_INSUFFICIENT_SRC; \
211 goto label_end_of_loop; \
218 /* Like ONE_MORE_BYTE, but 8-bit bytes of data at SRC are in multibyte
219 form if MULTIBYTEP is nonzero. */
221 #define ONE_MORE_BYTE_CHECK_MULTIBYTE(c1, multibytep) \
223 if (src >= src_end) \
225 coding->result = CODING_FINISH_INSUFFICIENT_SRC; \
226 goto label_end_of_loop; \
229 if (multibytep && c1 == LEADING_CODE_8_BIT_CONTROL) \
230 c1 = *src++ - 0x20; \
233 /* Set C to the next character at the source text pointed by `src'.
234 If there are not enough characters in the source, jump to
235 `label_end_of_loop'. The caller should set variables `coding'
236 `src', `src_end', and `translation_table' to appropriate pointers
237 in advance. This macro is used in encoding routines
238 `encode_coding_XXX', thus it assumes that the source text is in
239 multibyte form except for 8-bit characters. 8-bit characters are
240 in multibyte form if coding->src_multibyte is nonzero, else they
241 are represented by a single byte. */
243 #define ONE_MORE_CHAR(c) \
245 int len = src_end - src; \
249 coding->result = CODING_FINISH_INSUFFICIENT_SRC; \
250 goto label_end_of_loop; \
252 if (coding->src_multibyte \
253 || UNIBYTE_STR_AS_MULTIBYTE_P (src, len, bytes)) \
254 c = STRING_CHAR_AND_LENGTH (src, len, bytes); \
256 c = *src, bytes = 1; \
257 if (!NILP (translation_table)) \
258 c = translate_char (translation_table, c, -1, 0, 0); \
263 /* Produce a multibyte form of character C to `dst'. Jump to
264 `label_end_of_loop' if there's not enough space at `dst'.
266 If we are now in the middle of a composition sequence, the decoded
267 character may be ALTCHAR (for the current composition). In that
268 case, the character goes to coding->cmp_data->data instead of
271 This macro is used in decoding routines. */
273 #define EMIT_CHAR(c) \
275 if (! COMPOSING_P (coding) \
276 || coding->composing == COMPOSITION_RELATIVE \
277 || coding->composing == COMPOSITION_WITH_RULE) \
279 int bytes = CHAR_BYTES (c); \
280 if ((dst + bytes) > (dst_bytes ? dst_end : src)) \
282 coding->result = CODING_FINISH_INSUFFICIENT_DST; \
283 goto label_end_of_loop; \
285 dst += CHAR_STRING (c, dst); \
286 coding->produced_char++; \
289 if (COMPOSING_P (coding) \
290 && coding->composing != COMPOSITION_RELATIVE) \
292 CODING_ADD_COMPOSITION_COMPONENT (coding, c); \
293 coding->composition_rule_follows \
294 = coding->composing != COMPOSITION_WITH_ALTCHARS; \
299 #define EMIT_ONE_BYTE(c) \
301 if (dst >= (dst_bytes ? dst_end : src)) \
303 coding->result = CODING_FINISH_INSUFFICIENT_DST; \
304 goto label_end_of_loop; \
309 #define EMIT_TWO_BYTES(c1, c2) \
311 if (dst + 2 > (dst_bytes ? dst_end : src)) \
313 coding->result = CODING_FINISH_INSUFFICIENT_DST; \
314 goto label_end_of_loop; \
316 *dst++ = c1, *dst++ = c2; \
319 #define EMIT_BYTES(from, to) \
321 if (dst + (to - from) > (dst_bytes ? dst_end : src)) \
323 coding->result = CODING_FINISH_INSUFFICIENT_DST; \
324 goto label_end_of_loop; \
331 /*** 1. Preamble ***/
344 #include "composite.h"
349 #else /* not emacs */
353 #endif /* not emacs */
355 Lisp_Object Qcoding_system
, Qeol_type
;
356 Lisp_Object Qbuffer_file_coding_system
;
357 Lisp_Object Qpost_read_conversion
, Qpre_write_conversion
;
358 Lisp_Object Qno_conversion
, Qundecided
;
359 Lisp_Object Qcoding_system_history
;
360 Lisp_Object Qsafe_chars
;
361 Lisp_Object Qvalid_codes
;
363 extern Lisp_Object Qinsert_file_contents
, Qwrite_region
;
364 Lisp_Object Qcall_process
, Qcall_process_region
, Qprocess_argument
;
365 Lisp_Object Qstart_process
, Qopen_network_stream
;
366 Lisp_Object Qtarget_idx
;
368 Lisp_Object Vselect_safe_coding_system_function
;
370 /* Mnemonic string for each format of end-of-line. */
371 Lisp_Object eol_mnemonic_unix
, eol_mnemonic_dos
, eol_mnemonic_mac
;
372 /* Mnemonic string to indicate format of end-of-line is not yet
374 Lisp_Object eol_mnemonic_undecided
;
376 /* Format of end-of-line decided by system. This is CODING_EOL_LF on
377 Unix, CODING_EOL_CRLF on DOS/Windows, and CODING_EOL_CR on Mac. */
382 Lisp_Object Vcoding_system_list
, Vcoding_system_alist
;
384 Lisp_Object Qcoding_system_p
, Qcoding_system_error
;
386 /* Coding system emacs-mule and raw-text are for converting only
387 end-of-line format. */
388 Lisp_Object Qemacs_mule
, Qraw_text
;
390 /* Coding-systems are handed between Emacs Lisp programs and C internal
391 routines by the following three variables. */
392 /* Coding-system for reading files and receiving data from process. */
393 Lisp_Object Vcoding_system_for_read
;
394 /* Coding-system for writing files and sending data to process. */
395 Lisp_Object Vcoding_system_for_write
;
396 /* Coding-system actually used in the latest I/O. */
397 Lisp_Object Vlast_coding_system_used
;
399 /* A vector of length 256 which contains information about special
400 Latin codes (especially for dealing with Microsoft codes). */
401 Lisp_Object Vlatin_extra_code_table
;
403 /* Flag to inhibit code conversion of end-of-line format. */
404 int inhibit_eol_conversion
;
406 /* Flag to inhibit ISO2022 escape sequence detection. */
407 int inhibit_iso_escape_detection
;
409 /* Flag to make buffer-file-coding-system inherit from process-coding. */
410 int inherit_process_coding_system
;
412 /* Coding system to be used to encode text for terminal display. */
413 struct coding_system terminal_coding
;
415 /* Coding system to be used to encode text for terminal display when
416 terminal coding system is nil. */
417 struct coding_system safe_terminal_coding
;
419 /* Coding system of what is sent from terminal keyboard. */
420 struct coding_system keyboard_coding
;
422 /* Default coding system to be used to write a file. */
423 struct coding_system default_buffer_file_coding
;
425 Lisp_Object Vfile_coding_system_alist
;
426 Lisp_Object Vprocess_coding_system_alist
;
427 Lisp_Object Vnetwork_coding_system_alist
;
429 Lisp_Object Vlocale_coding_system
;
433 Lisp_Object Qcoding_category
, Qcoding_category_index
;
435 /* List of symbols `coding-category-xxx' ordered by priority. */
436 Lisp_Object Vcoding_category_list
;
438 /* Table of coding categories (Lisp symbols). */
439 Lisp_Object Vcoding_category_table
;
441 /* Table of names of symbol for each coding-category. */
442 char *coding_category_name
[CODING_CATEGORY_IDX_MAX
] = {
443 "coding-category-emacs-mule",
444 "coding-category-sjis",
445 "coding-category-iso-7",
446 "coding-category-iso-7-tight",
447 "coding-category-iso-8-1",
448 "coding-category-iso-8-2",
449 "coding-category-iso-7-else",
450 "coding-category-iso-8-else",
451 "coding-category-ccl",
452 "coding-category-big5",
453 "coding-category-utf-8",
454 "coding-category-utf-16-be",
455 "coding-category-utf-16-le",
456 "coding-category-raw-text",
457 "coding-category-binary"
460 /* Table of pointers to coding systems corresponding to each coding
462 struct coding_system
*coding_system_table
[CODING_CATEGORY_IDX_MAX
];
464 /* Table of coding category masks. Nth element is a mask for a coding
465 category of which priority is Nth. */
467 int coding_priorities
[CODING_CATEGORY_IDX_MAX
];
469 /* Flag to tell if we look up translation table on character code
471 Lisp_Object Venable_character_translation
;
472 /* Standard translation table to look up on decoding (reading). */
473 Lisp_Object Vstandard_translation_table_for_decode
;
474 /* Standard translation table to look up on encoding (writing). */
475 Lisp_Object Vstandard_translation_table_for_encode
;
477 Lisp_Object Qtranslation_table
;
478 Lisp_Object Qtranslation_table_id
;
479 Lisp_Object Qtranslation_table_for_decode
;
480 Lisp_Object Qtranslation_table_for_encode
;
482 /* Alist of charsets vs revision number. */
483 Lisp_Object Vcharset_revision_alist
;
485 /* Default coding systems used for process I/O. */
486 Lisp_Object Vdefault_process_coding_system
;
488 /* Global flag to tell that we can't call post-read-conversion and
489 pre-write-conversion functions. Usually the value is zero, but it
490 is set to 1 temporarily while such functions are running. This is
491 to avoid infinite recursive call. */
492 static int inhibit_pre_post_conversion
;
494 /* Char-table containing safe coding systems of each character. */
495 Lisp_Object Vchar_coding_system_table
;
496 Lisp_Object Qchar_coding_system
;
498 /* Return `safe-chars' property of coding system CODING. Don't check
499 validity of CODING. */
502 coding_safe_chars (coding
)
503 struct coding_system
*coding
;
505 Lisp_Object coding_spec
, plist
, safe_chars
;
507 coding_spec
= Fget (coding
->symbol
, Qcoding_system
);
508 plist
= XVECTOR (coding_spec
)->contents
[3];
509 safe_chars
= Fplist_get (XVECTOR (coding_spec
)->contents
[3], Qsafe_chars
);
510 return (CHAR_TABLE_P (safe_chars
) ? safe_chars
: Qt
);
513 #define CODING_SAFE_CHAR_P(safe_chars, c) \
514 (EQ (safe_chars, Qt) || !NILP (CHAR_TABLE_REF (safe_chars, c)))
517 /*** 2. Emacs internal format (emacs-mule) handlers ***/
519 /* Emacs' internal format for representation of multiple character
520 sets is a kind of multi-byte encoding, i.e. characters are
521 represented by variable-length sequences of one-byte codes.
523 ASCII characters and control characters (e.g. `tab', `newline') are
524 represented by one-byte sequences which are their ASCII codes, in
525 the range 0x00 through 0x7F.
527 8-bit characters of the range 0x80..0x9F are represented by
528 two-byte sequences of LEADING_CODE_8_BIT_CONTROL and (their 8-bit
531 8-bit characters of the range 0xA0..0xFF are represented by
532 one-byte sequences which are their 8-bit code.
534 The other characters are represented by a sequence of `base
535 leading-code', optional `extended leading-code', and one or two
536 `position-code's. The length of the sequence is determined by the
537 base leading-code. Leading-code takes the range 0x81 through 0x9D,
538 whereas extended leading-code and position-code take the range 0xA0
539 through 0xFF. See `charset.h' for more details about leading-code
542 --- CODE RANGE of Emacs' internal format ---
546 eight-bit-control LEADING_CODE_8_BIT_CONTROL + 0xA0..0xBF
547 eight-bit-graphic 0xA0..0xBF
548 ELSE 0x81..0x9D + [0xA0..0xFF]+
549 ---------------------------------------------
551 As this is the internal character representation, the format is
552 usually not used externally (i.e. in a file or in a data sent to a
553 process). But, it is possible to have a text externally in this
554 format (i.e. by encoding by the coding system `emacs-mule').
556 In that case, a sequence of one-byte codes has a slightly different
559 Firstly, all characters in eight-bit-control are represented by
560 one-byte sequences which are their 8-bit code.
562 Next, character composition data are represented by the byte
563 sequence of the form: 0x80 METHOD BYTES CHARS COMPONENT ...,
565 METHOD is 0xF0 plus one of composition method (enum
568 BYTES is 0xA0 plus the byte length of these composition data,
570 CHARS is 0xA0 plus the number of characters composed by these
573 COMPONENTs are characters of multibyte form or composition
574 rules encoded by two-byte of ASCII codes.
576 In addition, for backward compatibility, the following formats are
577 also recognized as composition data on decoding.
580 0x80 0xFF MSEQ RULE MSEQ RULE ... MSEQ
583 MSEQ is a multibyte form but in these special format:
584 ASCII: 0xA0 ASCII_CODE+0x80,
585 other: LEADING_CODE+0x20 FOLLOWING-BYTE ...,
586 RULE is a one byte code of the range 0xA0..0xF0 that
587 represents a composition rule.
590 enum emacs_code_class_type emacs_code_class
[256];
592 /* See the above "GENERAL NOTES on `detect_coding_XXX ()' functions".
593 Check if a text is encoded in Emacs' internal format. If it is,
594 return CODING_CATEGORY_MASK_EMACS_MULE, else return 0. */
597 detect_coding_emacs_mule (src
, src_end
, multibytep
)
598 unsigned char *src
, *src_end
;
603 /* Dummy for ONE_MORE_BYTE. */
604 struct coding_system dummy_coding
;
605 struct coding_system
*coding
= &dummy_coding
;
609 ONE_MORE_BYTE_CHECK_MULTIBYTE (c
, multibytep
);
617 ONE_MORE_BYTE_CHECK_MULTIBYTE (c
, multibytep
);
626 if (c
== ISO_CODE_ESC
|| c
== ISO_CODE_SI
|| c
== ISO_CODE_SO
)
629 else if (c
>= 0x80 && c
< 0xA0)
632 /* Old leading code for a composite character. */
636 unsigned char *src_base
= src
- 1;
639 if (!UNIBYTE_STR_AS_MULTIBYTE_P (src_base
, src_end
- src_base
,
642 src
= src_base
+ bytes
;
647 return CODING_CATEGORY_MASK_EMACS_MULE
;
651 /* Record the starting position START and METHOD of one composition. */
653 #define CODING_ADD_COMPOSITION_START(coding, start, method) \
655 struct composition_data *cmp_data = coding->cmp_data; \
656 int *data = cmp_data->data + cmp_data->used; \
657 coding->cmp_data_start = cmp_data->used; \
659 data[1] = cmp_data->char_offset + start; \
660 data[3] = (int) method; \
661 cmp_data->used += 4; \
664 /* Record the ending position END of the current composition. */
666 #define CODING_ADD_COMPOSITION_END(coding, end) \
668 struct composition_data *cmp_data = coding->cmp_data; \
669 int *data = cmp_data->data + coding->cmp_data_start; \
670 data[0] = cmp_data->used - coding->cmp_data_start; \
671 data[2] = cmp_data->char_offset + end; \
674 /* Record one COMPONENT (alternate character or composition rule). */
676 #define CODING_ADD_COMPOSITION_COMPONENT(coding, component) \
677 (coding->cmp_data->data[coding->cmp_data->used++] = component)
680 /* Get one byte from a data pointed by SRC and increment SRC. If SRC
681 is not less than SRC_END, return -1 without incrementing Src. */
683 #define SAFE_ONE_MORE_BYTE() (src >= src_end ? -1 : *src++)
686 /* Decode a character represented as a component of composition
687 sequence of Emacs 20 style at SRC. Set C to that character, store
688 its multibyte form sequence at P, and set P to the end of that
689 sequence. If no valid character is found, set C to -1. */
691 #define DECODE_EMACS_MULE_COMPOSITION_CHAR(c, p) \
695 c = SAFE_ONE_MORE_BYTE (); \
698 if (CHAR_HEAD_P (c)) \
700 else if (c == 0xA0) \
702 c = SAFE_ONE_MORE_BYTE (); \
711 else if (BASE_LEADING_CODE_P (c - 0x20)) \
713 unsigned char *p0 = p; \
717 bytes = BYTES_BY_CHAR_HEAD (c); \
720 c = SAFE_ONE_MORE_BYTE (); \
725 if (UNIBYTE_STR_AS_MULTIBYTE_P (p0, p - p0, bytes)) \
726 c = STRING_CHAR (p0, bytes); \
735 /* Decode a composition rule represented as a component of composition
736 sequence of Emacs 20 style at SRC. Set C to the rule. If not
737 valid rule is found, set C to -1. */
739 #define DECODE_EMACS_MULE_COMPOSITION_RULE(c) \
741 c = SAFE_ONE_MORE_BYTE (); \
743 if (c < 0 || c >= 81) \
747 gref = c / 9, nref = c % 9; \
748 c = COMPOSITION_ENCODE_RULE (gref, nref); \
753 /* Decode composition sequence encoded by `emacs-mule' at the source
754 pointed by SRC. SRC_END is the end of source. Store information
755 of the composition in CODING->cmp_data.
757 For backward compatibility, decode also a composition sequence of
758 Emacs 20 style. In that case, the composition sequence contains
759 characters that should be extracted into a buffer or string. Store
760 those characters at *DESTINATION in multibyte form.
762 If we encounter an invalid byte sequence, return 0.
763 If we encounter an insufficient source or destination, or
764 insufficient space in CODING->cmp_data, return 1.
765 Otherwise, return consumed bytes in the source.
769 decode_composition_emacs_mule (coding
, src
, src_end
,
770 destination
, dst_end
, dst_bytes
)
771 struct coding_system
*coding
;
772 unsigned char *src
, *src_end
, **destination
, *dst_end
;
775 unsigned char *dst
= *destination
;
776 int method
, data_len
, nchars
;
777 unsigned char *src_base
= src
++;
778 /* Store components of composition. */
779 int component
[COMPOSITION_DATA_MAX_BUNCH_LENGTH
];
781 /* Store multibyte form of characters to be composed. This is for
782 Emacs 20 style composition sequence. */
783 unsigned char buf
[MAX_COMPOSITION_COMPONENTS
* MAX_MULTIBYTE_LENGTH
];
784 unsigned char *bufp
= buf
;
785 int c
, i
, gref
, nref
;
787 if (coding
->cmp_data
->used
+ COMPOSITION_DATA_MAX_BUNCH_LENGTH
788 >= COMPOSITION_DATA_SIZE
)
790 coding
->result
= CODING_FINISH_INSUFFICIENT_CMP
;
795 if (c
- 0xF0 >= COMPOSITION_RELATIVE
796 && c
- 0xF0 <= COMPOSITION_WITH_RULE_ALTCHARS
)
801 with_rule
= (method
== COMPOSITION_WITH_RULE
802 || method
== COMPOSITION_WITH_RULE_ALTCHARS
);
806 || src_base
+ data_len
> src_end
)
812 for (ncomponent
= 0; src
< src_base
+ data_len
; ncomponent
++)
814 /* If it is longer than this, it can't be valid. */
815 if (ncomponent
>= COMPOSITION_DATA_MAX_BUNCH_LENGTH
)
818 if (ncomponent
% 2 && with_rule
)
820 ONE_MORE_BYTE (gref
);
822 ONE_MORE_BYTE (nref
);
824 c
= COMPOSITION_ENCODE_RULE (gref
, nref
);
829 if (UNIBYTE_STR_AS_MULTIBYTE_P (src
, src_end
- src
, bytes
))
830 c
= STRING_CHAR (src
, bytes
);
835 component
[ncomponent
] = c
;
840 /* This may be an old Emacs 20 style format. See the comment at
841 the section 2 of this file. */
842 while (src
< src_end
&& !CHAR_HEAD_P (*src
)) src
++;
844 && !(coding
->mode
& CODING_MODE_LAST_BLOCK
))
845 goto label_end_of_loop
;
851 method
= COMPOSITION_RELATIVE
;
852 for (ncomponent
= 0; ncomponent
< MAX_COMPOSITION_COMPONENTS
;)
854 DECODE_EMACS_MULE_COMPOSITION_CHAR (c
, bufp
);
857 component
[ncomponent
++] = c
;
865 method
= COMPOSITION_WITH_RULE
;
867 DECODE_EMACS_MULE_COMPOSITION_CHAR (c
, bufp
);
872 ncomponent
< MAX_COMPOSITION_COMPONENTS
* 2 - 1;)
874 DECODE_EMACS_MULE_COMPOSITION_RULE (c
);
877 component
[ncomponent
++] = c
;
878 DECODE_EMACS_MULE_COMPOSITION_CHAR (c
, bufp
);
881 component
[ncomponent
++] = c
;
885 nchars
= (ncomponent
+ 1) / 2;
891 if (buf
== bufp
|| dst
+ (bufp
- buf
) <= (dst_bytes
? dst_end
: src
))
893 CODING_ADD_COMPOSITION_START (coding
, coding
->produced_char
, method
);
894 for (i
= 0; i
< ncomponent
; i
++)
895 CODING_ADD_COMPOSITION_COMPONENT (coding
, component
[i
]);
896 CODING_ADD_COMPOSITION_END (coding
, coding
->produced_char
+ nchars
);
899 unsigned char *p
= buf
;
900 EMIT_BYTES (p
, bufp
);
901 *destination
+= bufp
- buf
;
902 coding
->produced_char
+= nchars
;
904 return (src
- src_base
);
910 /* See the above "GENERAL NOTES on `decode_coding_XXX ()' functions". */
913 decode_coding_emacs_mule (coding
, source
, destination
, src_bytes
, dst_bytes
)
914 struct coding_system
*coding
;
915 unsigned char *source
, *destination
;
916 int src_bytes
, dst_bytes
;
918 unsigned char *src
= source
;
919 unsigned char *src_end
= source
+ src_bytes
;
920 unsigned char *dst
= destination
;
921 unsigned char *dst_end
= destination
+ dst_bytes
;
922 /* SRC_BASE remembers the start position in source in each loop.
923 The loop will be exited when there's not enough source code, or
924 when there's not enough destination area to produce a
926 unsigned char *src_base
;
928 coding
->produced_char
= 0;
929 while ((src_base
= src
) < src_end
)
931 unsigned char tmp
[MAX_MULTIBYTE_LENGTH
], *p
;
938 if (coding
->eol_type
== CODING_EOL_CR
)
940 else if (coding
->eol_type
== CODING_EOL_CRLF
)
945 if (coding
->mode
& CODING_MODE_INHIBIT_INCONSISTENT_EOL
)
947 coding
->result
= CODING_FINISH_INCONSISTENT_EOL
;
948 goto label_end_of_loop
;
955 coding
->produced_char
++;
958 else if (*src
== '\n')
960 if ((coding
->eol_type
== CODING_EOL_CR
961 || coding
->eol_type
== CODING_EOL_CRLF
)
962 && coding
->mode
& CODING_MODE_INHIBIT_INCONSISTENT_EOL
)
964 coding
->result
= CODING_FINISH_INCONSISTENT_EOL
;
965 goto label_end_of_loop
;
968 coding
->produced_char
++;
971 else if (*src
== 0x80)
973 /* Start of composition data. */
974 int consumed
= decode_composition_emacs_mule (coding
, src
, src_end
,
978 goto label_end_of_loop
;
979 else if (consumed
> 0)
984 bytes
= CHAR_STRING (*src
, tmp
);
988 else if (UNIBYTE_STR_AS_MULTIBYTE_P (src
, src_end
- src
, bytes
))
995 bytes
= CHAR_STRING (*src
, tmp
);
999 if (dst
+ bytes
>= (dst_bytes
? dst_end
: src
))
1001 coding
->result
= CODING_FINISH_INSUFFICIENT_DST
;
1004 while (bytes
--) *dst
++ = *p
++;
1005 coding
->produced_char
++;
1008 coding
->consumed
= coding
->consumed_char
= src_base
- source
;
1009 coding
->produced
= dst
- destination
;
1013 /* Encode composition data stored at DATA into a special byte sequence
1014 starting by 0x80. Update CODING->cmp_data_start and maybe
1015 CODING->cmp_data for the next call. */
1017 #define ENCODE_COMPOSITION_EMACS_MULE(coding, data) \
1019 unsigned char buf[1024], *p0 = buf, *p; \
1020 int len = data[0]; \
1024 buf[1] = 0xF0 + data[3]; /* METHOD */ \
1025 buf[3] = 0xA0 + (data[2] - data[1]); /* COMPOSED-CHARS */ \
1027 if (data[3] == COMPOSITION_WITH_RULE \
1028 || data[3] == COMPOSITION_WITH_RULE_ALTCHARS) \
1030 p += CHAR_STRING (data[4], p); \
1031 for (i = 5; i < len; i += 2) \
1034 COMPOSITION_DECODE_RULE (data[i], gref, nref); \
1035 *p++ = 0x20 + gref; \
1036 *p++ = 0x20 + nref; \
1037 p += CHAR_STRING (data[i + 1], p); \
1042 for (i = 4; i < len; i++) \
1043 p += CHAR_STRING (data[i], p); \
1045 buf[2] = 0xA0 + (p - buf); /* COMPONENTS-BYTES */ \
1047 if (dst + (p - buf) + 4 > (dst_bytes ? dst_end : src)) \
1049 coding->result = CODING_FINISH_INSUFFICIENT_DST; \
1050 goto label_end_of_loop; \
1054 coding->cmp_data_start += data[0]; \
1055 if (coding->cmp_data_start == coding->cmp_data->used \
1056 && coding->cmp_data->next) \
1058 coding->cmp_data = coding->cmp_data->next; \
1059 coding->cmp_data_start = 0; \
1064 static void encode_eol
P_ ((struct coding_system
*, const unsigned char *,
1065 unsigned char *, int, int));
1068 encode_coding_emacs_mule (coding
, source
, destination
, src_bytes
, dst_bytes
)
1069 struct coding_system
*coding
;
1070 unsigned char *source
, *destination
;
1071 int src_bytes
, dst_bytes
;
1073 unsigned char *src
= source
;
1074 unsigned char *src_end
= source
+ src_bytes
;
1075 unsigned char *dst
= destination
;
1076 unsigned char *dst_end
= destination
+ dst_bytes
;
1077 unsigned char *src_base
;
1082 Lisp_Object translation_table
;
1084 translation_table
= Qnil
;
1086 /* Optimization for the case that there's no composition. */
1087 if (!coding
->cmp_data
|| coding
->cmp_data
->used
== 0)
1089 encode_eol (coding
, source
, destination
, src_bytes
, dst_bytes
);
1093 char_offset
= coding
->cmp_data
->char_offset
;
1094 data
= coding
->cmp_data
->data
+ coding
->cmp_data_start
;
1099 /* If SRC starts a composition, encode the information about the
1100 composition in advance. */
1101 if (coding
->cmp_data_start
< coding
->cmp_data
->used
1102 && char_offset
+ coding
->consumed_char
== data
[1])
1104 ENCODE_COMPOSITION_EMACS_MULE (coding
, data
);
1105 char_offset
= coding
->cmp_data
->char_offset
;
1106 data
= coding
->cmp_data
->data
+ coding
->cmp_data_start
;
1110 if (c
== '\n' && (coding
->eol_type
== CODING_EOL_CRLF
1111 || coding
->eol_type
== CODING_EOL_CR
))
1113 if (coding
->eol_type
== CODING_EOL_CRLF
)
1114 EMIT_TWO_BYTES ('\r', c
);
1116 EMIT_ONE_BYTE ('\r');
1118 else if (SINGLE_BYTE_CHAR_P (c
))
1121 EMIT_BYTES (src_base
, src
);
1122 coding
->consumed_char
++;
1125 coding
->consumed
= src_base
- source
;
1126 coding
->produced
= coding
->produced_char
= dst
- destination
;
1131 /*** 3. ISO2022 handlers ***/
1133 /* The following note describes the coding system ISO2022 briefly.
1134 Since the intention of this note is to help understand the
1135 functions in this file, some parts are NOT ACCURATE or are OVERLY
1136 SIMPLIFIED. For thorough understanding, please refer to the
1137 original document of ISO2022. This is equivalent to the standard
1138 ECMA-35, obtainable from <URL:http://www.ecma.ch/> (*).
1140 ISO2022 provides many mechanisms to encode several character sets
1141 in 7-bit and 8-bit environments. For 7-bit environments, all text
1142 is encoded using bytes less than 128. This may make the encoded
1143 text a little bit longer, but the text passes more easily through
1144 several types of gateway, some of which strip off the MSB (Most
1147 There are two kinds of character sets: control character sets and
1148 graphic character sets. The former contain control characters such
1149 as `newline' and `escape' to provide control functions (control
1150 functions are also provided by escape sequences). The latter
1151 contain graphic characters such as 'A' and '-'. Emacs recognizes
1152 two control character sets and many graphic character sets.
1154 Graphic character sets are classified into one of the following
1155 four classes, according to the number of bytes (DIMENSION) and
1156 number of characters in one dimension (CHARS) of the set:
1157 - DIMENSION1_CHARS94
1158 - DIMENSION1_CHARS96
1159 - DIMENSION2_CHARS94
1160 - DIMENSION2_CHARS96
1162 In addition, each character set is assigned an identification tag,
1163 unique for each set, called the "final character" (denoted as <F>
1164 hereafter). The <F> of each character set is decided by ECMA(*)
1165 when it is registered in ISO. The code range of <F> is 0x30..0x7F
1166 (0x30..0x3F are for private use only).
1168 Note (*): ECMA = European Computer Manufacturers Association
1170 Here are examples of graphic character sets [NAME(<F>)]:
1171 o DIMENSION1_CHARS94 -- ASCII('B'), right-half-of-JISX0201('I'), ...
1172 o DIMENSION1_CHARS96 -- right-half-of-ISO8859-1('A'), ...
1173 o DIMENSION2_CHARS94 -- GB2312('A'), JISX0208('B'), ...
1174 o DIMENSION2_CHARS96 -- none for the moment
1176 A code area (1 byte=8 bits) is divided into 4 areas, C0, GL, C1, and GR.
1177 C0 [0x00..0x1F] -- control character plane 0
1178 GL [0x20..0x7F] -- graphic character plane 0
1179 C1 [0x80..0x9F] -- control character plane 1
1180 GR [0xA0..0xFF] -- graphic character plane 1
1182 A control character set is directly designated and invoked to C0 or
1183 C1 by an escape sequence. The most common case is that:
1184 - ISO646's control character set is designated/invoked to C0, and
1185 - ISO6429's control character set is designated/invoked to C1,
1186 and usually these designations/invocations are omitted in encoded
1187 text. In a 7-bit environment, only C0 can be used, and a control
1188 character for C1 is encoded by an appropriate escape sequence to
1189 fit into the environment. All control characters for C1 are
1190 defined to have corresponding escape sequences.
1192 A graphic character set is at first designated to one of four
1193 graphic registers (G0 through G3), then these graphic registers are
1194 invoked to GL or GR. These designations and invocations can be
1195 done independently. The most common case is that G0 is invoked to
1196 GL, G1 is invoked to GR, and ASCII is designated to G0. Usually
1197 these invocations and designations are omitted in encoded text.
1198 In a 7-bit environment, only GL can be used.
1200 When a graphic character set of CHARS94 is invoked to GL, codes
1201 0x20 and 0x7F of the GL area work as control characters SPACE and
1202 DEL respectively, and codes 0xA0 and 0xFF of the GR area should not
1205 There are two ways of invocation: locking-shift and single-shift.
1206 With locking-shift, the invocation lasts until the next different
1207 invocation, whereas with single-shift, the invocation affects the
1208 following character only and doesn't affect the locking-shift
1209 state. Invocations are done by the following control characters or
1212 ----------------------------------------------------------------------
1213 abbrev function cntrl escape seq description
1214 ----------------------------------------------------------------------
1215 SI/LS0 (shift-in) 0x0F none invoke G0 into GL
1216 SO/LS1 (shift-out) 0x0E none invoke G1 into GL
1217 LS2 (locking-shift-2) none ESC 'n' invoke G2 into GL
1218 LS3 (locking-shift-3) none ESC 'o' invoke G3 into GL
1219 LS1R (locking-shift-1 right) none ESC '~' invoke G1 into GR (*)
1220 LS2R (locking-shift-2 right) none ESC '}' invoke G2 into GR (*)
1221 LS3R (locking-shift 3 right) none ESC '|' invoke G3 into GR (*)
1222 SS2 (single-shift-2) 0x8E ESC 'N' invoke G2 for one char
1223 SS3 (single-shift-3) 0x8F ESC 'O' invoke G3 for one char
1224 ----------------------------------------------------------------------
1225 (*) These are not used by any known coding system.
1227 Control characters for these functions are defined by macros
1228 ISO_CODE_XXX in `coding.h'.
1230 Designations are done by the following escape sequences:
1231 ----------------------------------------------------------------------
1232 escape sequence description
1233 ----------------------------------------------------------------------
1234 ESC '(' <F> designate DIMENSION1_CHARS94<F> to G0
1235 ESC ')' <F> designate DIMENSION1_CHARS94<F> to G1
1236 ESC '*' <F> designate DIMENSION1_CHARS94<F> to G2
1237 ESC '+' <F> designate DIMENSION1_CHARS94<F> to G3
1238 ESC ',' <F> designate DIMENSION1_CHARS96<F> to G0 (*)
1239 ESC '-' <F> designate DIMENSION1_CHARS96<F> to G1
1240 ESC '.' <F> designate DIMENSION1_CHARS96<F> to G2
1241 ESC '/' <F> designate DIMENSION1_CHARS96<F> to G3
1242 ESC '$' '(' <F> designate DIMENSION2_CHARS94<F> to G0 (**)
1243 ESC '$' ')' <F> designate DIMENSION2_CHARS94<F> to G1
1244 ESC '$' '*' <F> designate DIMENSION2_CHARS94<F> to G2
1245 ESC '$' '+' <F> designate DIMENSION2_CHARS94<F> to G3
1246 ESC '$' ',' <F> designate DIMENSION2_CHARS96<F> to G0 (*)
1247 ESC '$' '-' <F> designate DIMENSION2_CHARS96<F> to G1
1248 ESC '$' '.' <F> designate DIMENSION2_CHARS96<F> to G2
1249 ESC '$' '/' <F> designate DIMENSION2_CHARS96<F> to G3
1250 ----------------------------------------------------------------------
1252 In this list, "DIMENSION1_CHARS94<F>" means a graphic character set
1253 of dimension 1, chars 94, and final character <F>, etc...
1255 Note (*): Although these designations are not allowed in ISO2022,
1256 Emacs accepts them on decoding, and produces them on encoding
1257 CHARS96 character sets in a coding system which is characterized as
1258 7-bit environment, non-locking-shift, and non-single-shift.
1260 Note (**): If <F> is '@', 'A', or 'B', the intermediate character
1261 '(' can be omitted. We refer to this as "short-form" hereafter.
1263 Now you may notice that there are a lot of ways of encoding the
1264 same multilingual text in ISO2022. Actually, there exist many
1265 coding systems such as Compound Text (used in X11's inter client
1266 communication, ISO-2022-JP (used in Japanese Internet), ISO-2022-KR
1267 (used in Korean Internet), EUC (Extended UNIX Code, used in Asian
1268 localized platforms), and all of these are variants of ISO2022.
1270 In addition to the above, Emacs handles two more kinds of escape
1271 sequences: ISO6429's direction specification and Emacs' private
1272 sequence for specifying character composition.
1274 ISO6429's direction specification takes the following form:
1275 o CSI ']' -- end of the current direction
1276 o CSI '0' ']' -- end of the current direction
1277 o CSI '1' ']' -- start of left-to-right text
1278 o CSI '2' ']' -- start of right-to-left text
1279 The control character CSI (0x9B: control sequence introducer) is
1280 abbreviated to the escape sequence ESC '[' in a 7-bit environment.
1282 Character composition specification takes the following form:
1283 o ESC '0' -- start relative composition
1284 o ESC '1' -- end composition
1285 o ESC '2' -- start rule-base composition (*)
1286 o ESC '3' -- start relative composition with alternate chars (**)
1287 o ESC '4' -- start rule-base composition with alternate chars (**)
1288 Since these are not standard escape sequences of any ISO standard,
1289 the use of them with these meanings is restricted to Emacs only.
1291 (*) This form is used only in Emacs 20.5 and older versions,
1292 but the newer versions can safely decode it.
1293 (**) This form is used only in Emacs 21.1 and newer versions,
1294 and the older versions can't decode it.
1296 Here's a list of example usages of these composition escape
1297 sequences (categorized by `enum composition_method').
1299 COMPOSITION_RELATIVE:
1300 ESC 0 CHAR [ CHAR ] ESC 1
1301 COMPOSITION_WITH_RULE:
1302 ESC 2 CHAR [ RULE CHAR ] ESC 1
1303 COMPOSITION_WITH_ALTCHARS:
1304 ESC 3 ALTCHAR [ ALTCHAR ] ESC 0 CHAR [ CHAR ] ESC 1
1305 COMPOSITION_WITH_RULE_ALTCHARS:
1306 ESC 4 ALTCHAR [ RULE ALTCHAR ] ESC 0 CHAR [ CHAR ] ESC 1 */
1308 enum iso_code_class_type iso_code_class
[256];
1310 #define CHARSET_OK(idx, charset, c) \
1311 (coding_system_table[idx] \
1312 && (charset == CHARSET_ASCII \
1313 || (safe_chars = coding_safe_chars (coding_system_table[idx]), \
1314 CODING_SAFE_CHAR_P (safe_chars, c))) \
1315 && (CODING_SPEC_ISO_REQUESTED_DESIGNATION (coding_system_table[idx], \
1317 != CODING_SPEC_ISO_NO_REQUESTED_DESIGNATION))
1319 #define SHIFT_OUT_OK(idx) \
1320 (CODING_SPEC_ISO_INITIAL_DESIGNATION (coding_system_table[idx], 1) >= 0)
1322 /* See the above "GENERAL NOTES on `detect_coding_XXX ()' functions".
1323 Check if a text is encoded in ISO2022. If it is, return an
1324 integer in which appropriate flag bits any of:
1325 CODING_CATEGORY_MASK_ISO_7
1326 CODING_CATEGORY_MASK_ISO_7_TIGHT
1327 CODING_CATEGORY_MASK_ISO_8_1
1328 CODING_CATEGORY_MASK_ISO_8_2
1329 CODING_CATEGORY_MASK_ISO_7_ELSE
1330 CODING_CATEGORY_MASK_ISO_8_ELSE
1331 are set. If a code which should never appear in ISO2022 is found,
1335 detect_coding_iso2022 (src
, src_end
, multibytep
)
1336 unsigned char *src
, *src_end
;
1339 int mask
= CODING_CATEGORY_MASK_ISO
;
1341 int reg
[4], shift_out
= 0, single_shifting
= 0;
1343 /* Dummy for ONE_MORE_BYTE. */
1344 struct coding_system dummy_coding
;
1345 struct coding_system
*coding
= &dummy_coding
;
1346 Lisp_Object safe_chars
;
1348 reg
[0] = CHARSET_ASCII
, reg
[1] = reg
[2] = reg
[3] = -1;
1349 while (mask
&& src
< src_end
)
1351 ONE_MORE_BYTE_CHECK_MULTIBYTE (c
, multibytep
);
1356 if (inhibit_iso_escape_detection
)
1358 single_shifting
= 0;
1359 ONE_MORE_BYTE_CHECK_MULTIBYTE (c
, multibytep
);
1360 if (c
>= '(' && c
<= '/')
1362 /* Designation sequence for a charset of dimension 1. */
1363 ONE_MORE_BYTE_CHECK_MULTIBYTE (c1
, multibytep
);
1364 if (c1
< ' ' || c1
>= 0x80
1365 || (charset
= iso_charset_table
[0][c
>= ','][c1
]) < 0)
1366 /* Invalid designation sequence. Just ignore. */
1368 reg
[(c
- '(') % 4] = charset
;
1372 /* Designation sequence for a charset of dimension 2. */
1373 ONE_MORE_BYTE_CHECK_MULTIBYTE (c
, multibytep
);
1374 if (c
>= '@' && c
<= 'B')
1375 /* Designation for JISX0208.1978, GB2312, or JISX0208. */
1376 reg
[0] = charset
= iso_charset_table
[1][0][c
];
1377 else if (c
>= '(' && c
<= '/')
1379 ONE_MORE_BYTE_CHECK_MULTIBYTE (c1
, multibytep
);
1380 if (c1
< ' ' || c1
>= 0x80
1381 || (charset
= iso_charset_table
[1][c
>= ','][c1
]) < 0)
1382 /* Invalid designation sequence. Just ignore. */
1384 reg
[(c
- '(') % 4] = charset
;
1387 /* Invalid designation sequence. Just ignore. */
1390 else if (c
== 'N' || c
== 'O')
1392 /* ESC <Fe> for SS2 or SS3. */
1393 mask
&= CODING_CATEGORY_MASK_ISO_7_ELSE
;
1396 else if (c
>= '0' && c
<= '4')
1398 /* ESC <Fp> for start/end composition. */
1399 mask_found
|= CODING_CATEGORY_MASK_ISO
;
1403 /* Invalid escape sequence. Just ignore. */
1406 /* We found a valid designation sequence for CHARSET. */
1407 mask
&= ~CODING_CATEGORY_MASK_ISO_8BIT
;
1408 c
= MAKE_CHAR (charset
, 0, 0);
1409 if (CHARSET_OK (CODING_CATEGORY_IDX_ISO_7
, charset
, c
))
1410 mask_found
|= CODING_CATEGORY_MASK_ISO_7
;
1412 mask
&= ~CODING_CATEGORY_MASK_ISO_7
;
1413 if (CHARSET_OK (CODING_CATEGORY_IDX_ISO_7_TIGHT
, charset
, c
))
1414 mask_found
|= CODING_CATEGORY_MASK_ISO_7_TIGHT
;
1416 mask
&= ~CODING_CATEGORY_MASK_ISO_7_TIGHT
;
1417 if (CHARSET_OK (CODING_CATEGORY_IDX_ISO_7_ELSE
, charset
, c
))
1418 mask_found
|= CODING_CATEGORY_MASK_ISO_7_ELSE
;
1420 mask
&= ~CODING_CATEGORY_MASK_ISO_7_ELSE
;
1421 if (CHARSET_OK (CODING_CATEGORY_IDX_ISO_8_ELSE
, charset
, c
))
1422 mask_found
|= CODING_CATEGORY_MASK_ISO_8_ELSE
;
1424 mask
&= ~CODING_CATEGORY_MASK_ISO_8_ELSE
;
1428 if (inhibit_iso_escape_detection
)
1430 single_shifting
= 0;
1433 || SHIFT_OUT_OK (CODING_CATEGORY_IDX_ISO_7_ELSE
)
1434 || SHIFT_OUT_OK (CODING_CATEGORY_IDX_ISO_8_ELSE
)))
1436 /* Locking shift out. */
1437 mask
&= ~CODING_CATEGORY_MASK_ISO_7BIT
;
1438 mask_found
|= CODING_CATEGORY_MASK_ISO_SHIFT
;
1443 if (inhibit_iso_escape_detection
)
1445 single_shifting
= 0;
1448 /* Locking shift in. */
1449 mask
&= ~CODING_CATEGORY_MASK_ISO_7BIT
;
1450 mask_found
|= CODING_CATEGORY_MASK_ISO_SHIFT
;
1455 single_shifting
= 0;
1459 int newmask
= CODING_CATEGORY_MASK_ISO_8_ELSE
;
1461 if (inhibit_iso_escape_detection
)
1463 if (c
!= ISO_CODE_CSI
)
1465 if (coding_system_table
[CODING_CATEGORY_IDX_ISO_8_1
]->flags
1466 & CODING_FLAG_ISO_SINGLE_SHIFT
)
1467 newmask
|= CODING_CATEGORY_MASK_ISO_8_1
;
1468 if (coding_system_table
[CODING_CATEGORY_IDX_ISO_8_2
]->flags
1469 & CODING_FLAG_ISO_SINGLE_SHIFT
)
1470 newmask
|= CODING_CATEGORY_MASK_ISO_8_2
;
1471 single_shifting
= 1;
1473 if (VECTORP (Vlatin_extra_code_table
)
1474 && !NILP (XVECTOR (Vlatin_extra_code_table
)->contents
[c
]))
1476 if (coding_system_table
[CODING_CATEGORY_IDX_ISO_8_1
]->flags
1477 & CODING_FLAG_ISO_LATIN_EXTRA
)
1478 newmask
|= CODING_CATEGORY_MASK_ISO_8_1
;
1479 if (coding_system_table
[CODING_CATEGORY_IDX_ISO_8_2
]->flags
1480 & CODING_FLAG_ISO_LATIN_EXTRA
)
1481 newmask
|= CODING_CATEGORY_MASK_ISO_8_2
;
1484 mask_found
|= newmask
;
1491 single_shifting
= 0;
1496 single_shifting
= 0;
1497 if (VECTORP (Vlatin_extra_code_table
)
1498 && !NILP (XVECTOR (Vlatin_extra_code_table
)->contents
[c
]))
1502 if (coding_system_table
[CODING_CATEGORY_IDX_ISO_8_1
]->flags
1503 & CODING_FLAG_ISO_LATIN_EXTRA
)
1504 newmask
|= CODING_CATEGORY_MASK_ISO_8_1
;
1505 if (coding_system_table
[CODING_CATEGORY_IDX_ISO_8_2
]->flags
1506 & CODING_FLAG_ISO_LATIN_EXTRA
)
1507 newmask
|= CODING_CATEGORY_MASK_ISO_8_2
;
1509 mask_found
|= newmask
;
1516 mask
&= ~(CODING_CATEGORY_MASK_ISO_7BIT
1517 | CODING_CATEGORY_MASK_ISO_7_ELSE
);
1518 mask_found
|= CODING_CATEGORY_MASK_ISO_8_1
;
1519 /* Check the length of succeeding codes of the range
1520 0xA0..0FF. If the byte length is odd, we exclude
1521 CODING_CATEGORY_MASK_ISO_8_2. We can check this only
1522 when we are not single shifting. */
1523 if (!single_shifting
1524 && mask
& CODING_CATEGORY_MASK_ISO_8_2
)
1529 while (src
< src_end
)
1531 ONE_MORE_BYTE_CHECK_MULTIBYTE (c
, multibytep
);
1537 if (i
& 1 && src
< src_end
)
1538 mask
&= ~CODING_CATEGORY_MASK_ISO_8_2
;
1540 mask_found
|= CODING_CATEGORY_MASK_ISO_8_2
;
1542 /* This means that we have read one extra byte. */
1550 return (mask
& mask_found
);
1553 /* Decode a character of which charset is CHARSET, the 1st position
1554 code is C1, the 2nd position code is C2, and return the decoded
1555 character code. If the variable `translation_table' is non-nil,
1556 returned the translated code. */
1558 #define DECODE_ISO_CHARACTER(charset, c1, c2) \
1559 (NILP (translation_table) \
1560 ? MAKE_CHAR (charset, c1, c2) \
1561 : translate_char (translation_table, -1, charset, c1, c2))
1563 /* Set designation state into CODING. */
1564 #define DECODE_DESIGNATION(reg, dimension, chars, final_char) \
1568 if (final_char < '0' || final_char >= 128) \
1569 goto label_invalid_code; \
1570 charset = ISO_CHARSET_TABLE (make_number (dimension), \
1571 make_number (chars), \
1572 make_number (final_char)); \
1573 c = MAKE_CHAR (charset, 0, 0); \
1575 && (CODING_SPEC_ISO_REQUESTED_DESIGNATION (coding, charset) == reg \
1576 || CODING_SAFE_CHAR_P (safe_chars, c))) \
1578 if (coding->spec.iso2022.last_invalid_designation_register == 0 \
1580 && charset == CHARSET_ASCII) \
1582 /* We should insert this designation sequence as is so \
1583 that it is surely written back to a file. */ \
1584 coding->spec.iso2022.last_invalid_designation_register = -1; \
1585 goto label_invalid_code; \
1587 coding->spec.iso2022.last_invalid_designation_register = -1; \
1588 if ((coding->mode & CODING_MODE_DIRECTION) \
1589 && CHARSET_REVERSE_CHARSET (charset) >= 0) \
1590 charset = CHARSET_REVERSE_CHARSET (charset); \
1591 CODING_SPEC_ISO_DESIGNATION (coding, reg) = charset; \
1595 coding->spec.iso2022.last_invalid_designation_register = reg; \
1596 goto label_invalid_code; \
1600 /* Allocate a memory block for storing information about compositions.
1601 The block is chained to the already allocated blocks. */
1604 coding_allocate_composition_data (coding
, char_offset
)
1605 struct coding_system
*coding
;
1608 struct composition_data
*cmp_data
1609 = (struct composition_data
*) xmalloc (sizeof *cmp_data
);
1611 cmp_data
->char_offset
= char_offset
;
1613 cmp_data
->prev
= coding
->cmp_data
;
1614 cmp_data
->next
= NULL
;
1615 if (coding
->cmp_data
)
1616 coding
->cmp_data
->next
= cmp_data
;
1617 coding
->cmp_data
= cmp_data
;
1618 coding
->cmp_data_start
= 0;
1621 /* Handle composition start sequence ESC 0, ESC 2, ESC 3, or ESC 4.
1622 ESC 0 : relative composition : ESC 0 CHAR ... ESC 1
1623 ESC 2 : rulebase composition : ESC 2 CHAR RULE CHAR RULE ... CHAR ESC 1
1624 ESC 3 : altchar composition : ESC 3 ALT ... ESC 0 CHAR ... ESC 1
1625 ESC 4 : alt&rule composition : ESC 4 ALT RULE .. ALT ESC 0 CHAR ... ESC 1
1628 #define DECODE_COMPOSITION_START(c1) \
1630 if (coding->composing == COMPOSITION_DISABLED) \
1632 *dst++ = ISO_CODE_ESC; \
1633 *dst++ = c1 & 0x7f; \
1634 coding->produced_char += 2; \
1636 else if (!COMPOSING_P (coding)) \
1638 /* This is surely the start of a composition. We must be sure \
1639 that coding->cmp_data has enough space to store the \
1640 information about the composition. If not, terminate the \
1641 current decoding loop, allocate one more memory block for \
1642 coding->cmp_data in the caller, then start the decoding \
1643 loop again. We can't allocate memory here directly because \
1644 it may cause buffer/string relocation. */ \
1645 if (!coding->cmp_data \
1646 || (coding->cmp_data->used + COMPOSITION_DATA_MAX_BUNCH_LENGTH \
1647 >= COMPOSITION_DATA_SIZE)) \
1649 coding->result = CODING_FINISH_INSUFFICIENT_CMP; \
1650 goto label_end_of_loop; \
1652 coding->composing = (c1 == '0' ? COMPOSITION_RELATIVE \
1653 : c1 == '2' ? COMPOSITION_WITH_RULE \
1654 : c1 == '3' ? COMPOSITION_WITH_ALTCHARS \
1655 : COMPOSITION_WITH_RULE_ALTCHARS); \
1656 CODING_ADD_COMPOSITION_START (coding, coding->produced_char, \
1657 coding->composing); \
1658 coding->composition_rule_follows = 0; \
1662 /* We are already handling a composition. If the method is \
1663 the following two, the codes following the current escape \
1664 sequence are actual characters stored in a buffer. */ \
1665 if (coding->composing == COMPOSITION_WITH_ALTCHARS \
1666 || coding->composing == COMPOSITION_WITH_RULE_ALTCHARS) \
1668 coding->composing = COMPOSITION_RELATIVE; \
1669 coding->composition_rule_follows = 0; \
1674 /* Handle composition end sequence ESC 1. */
1676 #define DECODE_COMPOSITION_END(c1) \
1678 if (! COMPOSING_P (coding)) \
1680 *dst++ = ISO_CODE_ESC; \
1682 coding->produced_char += 2; \
1686 CODING_ADD_COMPOSITION_END (coding, coding->produced_char); \
1687 coding->composing = COMPOSITION_NO; \
1691 /* Decode a composition rule from the byte C1 (and maybe one more byte
1692 from SRC) and store one encoded composition rule in
1693 coding->cmp_data. */
1695 #define DECODE_COMPOSITION_RULE(c1) \
1699 if (c1 < 81) /* old format (before ver.21) */ \
1701 int gref = (c1) / 9; \
1702 int nref = (c1) % 9; \
1703 if (gref == 4) gref = 10; \
1704 if (nref == 4) nref = 10; \
1705 rule = COMPOSITION_ENCODE_RULE (gref, nref); \
1707 else if (c1 < 93) /* new format (after ver.21) */ \
1709 ONE_MORE_BYTE (c2); \
1710 rule = COMPOSITION_ENCODE_RULE (c1 - 81, c2 - 32); \
1712 CODING_ADD_COMPOSITION_COMPONENT (coding, rule); \
1713 coding->composition_rule_follows = 0; \
1717 /* See the above "GENERAL NOTES on `decode_coding_XXX ()' functions". */
1720 decode_coding_iso2022 (coding
, source
, destination
, src_bytes
, dst_bytes
)
1721 struct coding_system
*coding
;
1722 unsigned char *source
, *destination
;
1723 int src_bytes
, dst_bytes
;
1725 unsigned char *src
= source
;
1726 unsigned char *src_end
= source
+ src_bytes
;
1727 unsigned char *dst
= destination
;
1728 unsigned char *dst_end
= destination
+ dst_bytes
;
1729 /* Charsets invoked to graphic plane 0 and 1 respectively. */
1730 int charset0
= CODING_SPEC_ISO_PLANE_CHARSET (coding
, 0);
1731 int charset1
= CODING_SPEC_ISO_PLANE_CHARSET (coding
, 1);
1732 /* SRC_BASE remembers the start position in source in each loop.
1733 The loop will be exited when there's not enough source code
1734 (within macro ONE_MORE_BYTE), or when there's not enough
1735 destination area to produce a character (within macro
1737 unsigned char *src_base
;
1739 Lisp_Object translation_table
;
1740 Lisp_Object safe_chars
;
1742 safe_chars
= coding_safe_chars (coding
);
1744 if (NILP (Venable_character_translation
))
1745 translation_table
= Qnil
;
1748 translation_table
= coding
->translation_table_for_decode
;
1749 if (NILP (translation_table
))
1750 translation_table
= Vstandard_translation_table_for_decode
;
1753 coding
->result
= CODING_FINISH_NORMAL
;
1762 /* We produce no character or one character. */
1763 switch (iso_code_class
[c1
])
1765 case ISO_0x20_or_0x7F
:
1766 if (COMPOSING_P (coding
) && coding
->composition_rule_follows
)
1768 DECODE_COMPOSITION_RULE (c1
);
1771 if (charset0
< 0 || CHARSET_CHARS (charset0
) == 94)
1773 /* This is SPACE or DEL. */
1774 charset
= CHARSET_ASCII
;
1777 /* This is a graphic character, we fall down ... */
1779 case ISO_graphic_plane_0
:
1780 if (COMPOSING_P (coding
) && coding
->composition_rule_follows
)
1782 DECODE_COMPOSITION_RULE (c1
);
1788 case ISO_0xA0_or_0xFF
:
1789 if (charset1
< 0 || CHARSET_CHARS (charset1
) == 94
1790 || coding
->flags
& CODING_FLAG_ISO_SEVEN_BITS
)
1791 goto label_invalid_code
;
1792 /* This is a graphic character, we fall down ... */
1794 case ISO_graphic_plane_1
:
1796 goto label_invalid_code
;
1801 if (COMPOSING_P (coding
))
1802 DECODE_COMPOSITION_END ('1');
1804 /* All ISO2022 control characters in this class have the
1805 same representation in Emacs internal format. */
1807 && (coding
->mode
& CODING_MODE_INHIBIT_INCONSISTENT_EOL
)
1808 && (coding
->eol_type
== CODING_EOL_CR
1809 || coding
->eol_type
== CODING_EOL_CRLF
))
1811 coding
->result
= CODING_FINISH_INCONSISTENT_EOL
;
1812 goto label_end_of_loop
;
1814 charset
= CHARSET_ASCII
;
1818 if (COMPOSING_P (coding
))
1819 DECODE_COMPOSITION_END ('1');
1820 goto label_invalid_code
;
1822 case ISO_carriage_return
:
1823 if (COMPOSING_P (coding
))
1824 DECODE_COMPOSITION_END ('1');
1826 if (coding
->eol_type
== CODING_EOL_CR
)
1828 else if (coding
->eol_type
== CODING_EOL_CRLF
)
1831 if (c1
!= ISO_CODE_LF
)
1833 if (coding
->mode
& CODING_MODE_INHIBIT_INCONSISTENT_EOL
)
1835 coding
->result
= CODING_FINISH_INCONSISTENT_EOL
;
1836 goto label_end_of_loop
;
1842 charset
= CHARSET_ASCII
;
1846 if (! (coding
->flags
& CODING_FLAG_ISO_LOCKING_SHIFT
)
1847 || CODING_SPEC_ISO_DESIGNATION (coding
, 1) < 0)
1848 goto label_invalid_code
;
1849 CODING_SPEC_ISO_INVOCATION (coding
, 0) = 1;
1850 charset0
= CODING_SPEC_ISO_PLANE_CHARSET (coding
, 0);
1854 if (! (coding
->flags
& CODING_FLAG_ISO_LOCKING_SHIFT
))
1855 goto label_invalid_code
;
1856 CODING_SPEC_ISO_INVOCATION (coding
, 0) = 0;
1857 charset0
= CODING_SPEC_ISO_PLANE_CHARSET (coding
, 0);
1860 case ISO_single_shift_2_7
:
1861 case ISO_single_shift_2
:
1862 if (! (coding
->flags
& CODING_FLAG_ISO_SINGLE_SHIFT
))
1863 goto label_invalid_code
;
1864 /* SS2 is handled as an escape sequence of ESC 'N' */
1866 goto label_escape_sequence
;
1868 case ISO_single_shift_3
:
1869 if (! (coding
->flags
& CODING_FLAG_ISO_SINGLE_SHIFT
))
1870 goto label_invalid_code
;
1871 /* SS2 is handled as an escape sequence of ESC 'O' */
1873 goto label_escape_sequence
;
1875 case ISO_control_sequence_introducer
:
1876 /* CSI is handled as an escape sequence of ESC '[' ... */
1878 goto label_escape_sequence
;
1882 label_escape_sequence
:
1883 /* Escape sequences handled by Emacs are invocation,
1884 designation, direction specification, and character
1885 composition specification. */
1888 case '&': /* revision of following character set */
1890 if (!(c1
>= '@' && c1
<= '~'))
1891 goto label_invalid_code
;
1893 if (c1
!= ISO_CODE_ESC
)
1894 goto label_invalid_code
;
1896 goto label_escape_sequence
;
1898 case '$': /* designation of 2-byte character set */
1899 if (! (coding
->flags
& CODING_FLAG_ISO_DESIGNATION
))
1900 goto label_invalid_code
;
1902 if (c1
>= '@' && c1
<= 'B')
1903 { /* designation of JISX0208.1978, GB2312.1980,
1905 DECODE_DESIGNATION (0, 2, 94, c1
);
1907 else if (c1
>= 0x28 && c1
<= 0x2B)
1908 { /* designation of DIMENSION2_CHARS94 character set */
1910 DECODE_DESIGNATION (c1
- 0x28, 2, 94, c2
);
1912 else if (c1
>= 0x2C && c1
<= 0x2F)
1913 { /* designation of DIMENSION2_CHARS96 character set */
1915 DECODE_DESIGNATION (c1
- 0x2C, 2, 96, c2
);
1918 goto label_invalid_code
;
1919 /* We must update these variables now. */
1920 charset0
= CODING_SPEC_ISO_PLANE_CHARSET (coding
, 0);
1921 charset1
= CODING_SPEC_ISO_PLANE_CHARSET (coding
, 1);
1924 case 'n': /* invocation of locking-shift-2 */
1925 if (! (coding
->flags
& CODING_FLAG_ISO_LOCKING_SHIFT
)
1926 || CODING_SPEC_ISO_DESIGNATION (coding
, 2) < 0)
1927 goto label_invalid_code
;
1928 CODING_SPEC_ISO_INVOCATION (coding
, 0) = 2;
1929 charset0
= CODING_SPEC_ISO_PLANE_CHARSET (coding
, 0);
1932 case 'o': /* invocation of locking-shift-3 */
1933 if (! (coding
->flags
& CODING_FLAG_ISO_LOCKING_SHIFT
)
1934 || CODING_SPEC_ISO_DESIGNATION (coding
, 3) < 0)
1935 goto label_invalid_code
;
1936 CODING_SPEC_ISO_INVOCATION (coding
, 0) = 3;
1937 charset0
= CODING_SPEC_ISO_PLANE_CHARSET (coding
, 0);
1940 case 'N': /* invocation of single-shift-2 */
1941 if (! (coding
->flags
& CODING_FLAG_ISO_SINGLE_SHIFT
)
1942 || CODING_SPEC_ISO_DESIGNATION (coding
, 2) < 0)
1943 goto label_invalid_code
;
1944 charset
= CODING_SPEC_ISO_DESIGNATION (coding
, 2);
1946 if (c1
< 0x20 || (c1
>= 0x80 && c1
< 0xA0))
1947 goto label_invalid_code
;
1950 case 'O': /* invocation of single-shift-3 */
1951 if (! (coding
->flags
& CODING_FLAG_ISO_SINGLE_SHIFT
)
1952 || CODING_SPEC_ISO_DESIGNATION (coding
, 3) < 0)
1953 goto label_invalid_code
;
1954 charset
= CODING_SPEC_ISO_DESIGNATION (coding
, 3);
1956 if (c1
< 0x20 || (c1
>= 0x80 && c1
< 0xA0))
1957 goto label_invalid_code
;
1960 case '0': case '2': case '3': case '4': /* start composition */
1961 DECODE_COMPOSITION_START (c1
);
1964 case '1': /* end composition */
1965 DECODE_COMPOSITION_END (c1
);
1968 case '[': /* specification of direction */
1969 if (coding
->flags
& CODING_FLAG_ISO_NO_DIRECTION
)
1970 goto label_invalid_code
;
1971 /* For the moment, nested direction is not supported.
1972 So, `coding->mode & CODING_MODE_DIRECTION' zero means
1973 left-to-right, and nonzero means right-to-left. */
1977 case ']': /* end of the current direction */
1978 coding
->mode
&= ~CODING_MODE_DIRECTION
;
1980 case '0': /* end of the current direction */
1981 case '1': /* start of left-to-right direction */
1984 coding
->mode
&= ~CODING_MODE_DIRECTION
;
1986 goto label_invalid_code
;
1989 case '2': /* start of right-to-left direction */
1992 coding
->mode
|= CODING_MODE_DIRECTION
;
1994 goto label_invalid_code
;
1998 goto label_invalid_code
;
2003 if (! (coding
->flags
& CODING_FLAG_ISO_DESIGNATION
))
2004 goto label_invalid_code
;
2005 if (c1
>= 0x28 && c1
<= 0x2B)
2006 { /* designation of DIMENSION1_CHARS94 character set */
2008 DECODE_DESIGNATION (c1
- 0x28, 1, 94, c2
);
2010 else if (c1
>= 0x2C && c1
<= 0x2F)
2011 { /* designation of DIMENSION1_CHARS96 character set */
2013 DECODE_DESIGNATION (c1
- 0x2C, 1, 96, c2
);
2016 goto label_invalid_code
;
2017 /* We must update these variables now. */
2018 charset0
= CODING_SPEC_ISO_PLANE_CHARSET (coding
, 0);
2019 charset1
= CODING_SPEC_ISO_PLANE_CHARSET (coding
, 1);
2024 /* Now we know CHARSET and 1st position code C1 of a character.
2025 Produce a multibyte sequence for that character while getting
2026 2nd position code C2 if necessary. */
2027 if (CHARSET_DIMENSION (charset
) == 2)
2030 if (c1
< 0x80 ? c2
< 0x20 || c2
>= 0x80 : c2
< 0xA0)
2031 /* C2 is not in a valid range. */
2032 goto label_invalid_code
;
2034 c
= DECODE_ISO_CHARACTER (charset
, c1
, c2
);
2040 if (COMPOSING_P (coding
))
2041 DECODE_COMPOSITION_END ('1');
2048 coding
->consumed
= coding
->consumed_char
= src_base
- source
;
2049 coding
->produced
= dst
- destination
;
2054 /* ISO2022 encoding stuff. */
2057 It is not enough to say just "ISO2022" on encoding, we have to
2058 specify more details. In Emacs, each ISO2022 coding system
2059 variant has the following specifications:
2060 1. Initial designation to G0 through G3.
2061 2. Allows short-form designation?
2062 3. ASCII should be designated to G0 before control characters?
2063 4. ASCII should be designated to G0 at end of line?
2064 5. 7-bit environment or 8-bit environment?
2065 6. Use locking-shift?
2066 7. Use Single-shift?
2067 And the following two are only for Japanese:
2068 8. Use ASCII in place of JIS0201-1976-Roman?
2069 9. Use JISX0208-1983 in place of JISX0208-1978?
2070 These specifications are encoded in `coding->flags' as flag bits
2071 defined by macros CODING_FLAG_ISO_XXX. See `coding.h' for more
2075 /* Produce codes (escape sequence) for designating CHARSET to graphic
2076 register REG at DST, and increment DST. If <final-char> of CHARSET is
2077 '@', 'A', or 'B' and the coding system CODING allows, produce
2078 designation sequence of short-form. */
2080 #define ENCODE_DESIGNATION(charset, reg, coding) \
2082 unsigned char final_char = CHARSET_ISO_FINAL_CHAR (charset); \
2083 char *intermediate_char_94 = "()*+"; \
2084 char *intermediate_char_96 = ",-./"; \
2085 int revision = CODING_SPEC_ISO_REVISION_NUMBER(coding, charset); \
2087 if (revision < 255) \
2089 *dst++ = ISO_CODE_ESC; \
2091 *dst++ = '@' + revision; \
2093 *dst++ = ISO_CODE_ESC; \
2094 if (CHARSET_DIMENSION (charset) == 1) \
2096 if (CHARSET_CHARS (charset) == 94) \
2097 *dst++ = (unsigned char) (intermediate_char_94[reg]); \
2099 *dst++ = (unsigned char) (intermediate_char_96[reg]); \
2104 if (CHARSET_CHARS (charset) == 94) \
2106 if (! (coding->flags & CODING_FLAG_ISO_SHORT_FORM) \
2108 || final_char < '@' || final_char > 'B') \
2109 *dst++ = (unsigned char) (intermediate_char_94[reg]); \
2112 *dst++ = (unsigned char) (intermediate_char_96[reg]); \
2114 *dst++ = final_char; \
2115 CODING_SPEC_ISO_DESIGNATION (coding, reg) = charset; \
2118 /* The following two macros produce codes (control character or escape
2119 sequence) for ISO2022 single-shift functions (single-shift-2 and
2122 #define ENCODE_SINGLE_SHIFT_2 \
2124 if (coding->flags & CODING_FLAG_ISO_SEVEN_BITS) \
2125 *dst++ = ISO_CODE_ESC, *dst++ = 'N'; \
2127 *dst++ = ISO_CODE_SS2; \
2128 CODING_SPEC_ISO_SINGLE_SHIFTING (coding) = 1; \
2131 #define ENCODE_SINGLE_SHIFT_3 \
2133 if (coding->flags & CODING_FLAG_ISO_SEVEN_BITS) \
2134 *dst++ = ISO_CODE_ESC, *dst++ = 'O'; \
2136 *dst++ = ISO_CODE_SS3; \
2137 CODING_SPEC_ISO_SINGLE_SHIFTING (coding) = 1; \
2140 /* The following four macros produce codes (control character or
2141 escape sequence) for ISO2022 locking-shift functions (shift-in,
2142 shift-out, locking-shift-2, and locking-shift-3). */
2144 #define ENCODE_SHIFT_IN \
2146 *dst++ = ISO_CODE_SI; \
2147 CODING_SPEC_ISO_INVOCATION (coding, 0) = 0; \
2150 #define ENCODE_SHIFT_OUT \
2152 *dst++ = ISO_CODE_SO; \
2153 CODING_SPEC_ISO_INVOCATION (coding, 0) = 1; \
2156 #define ENCODE_LOCKING_SHIFT_2 \
2158 *dst++ = ISO_CODE_ESC, *dst++ = 'n'; \
2159 CODING_SPEC_ISO_INVOCATION (coding, 0) = 2; \
2162 #define ENCODE_LOCKING_SHIFT_3 \
2164 *dst++ = ISO_CODE_ESC, *dst++ = 'o'; \
2165 CODING_SPEC_ISO_INVOCATION (coding, 0) = 3; \
2168 /* Produce codes for a DIMENSION1 character whose character set is
2169 CHARSET and whose position-code is C1. Designation and invocation
2170 sequences are also produced in advance if necessary. */
2172 #define ENCODE_ISO_CHARACTER_DIMENSION1(charset, c1) \
2174 if (CODING_SPEC_ISO_SINGLE_SHIFTING (coding)) \
2176 if (coding->flags & CODING_FLAG_ISO_SEVEN_BITS) \
2177 *dst++ = c1 & 0x7F; \
2179 *dst++ = c1 | 0x80; \
2180 CODING_SPEC_ISO_SINGLE_SHIFTING (coding) = 0; \
2183 else if (charset == CODING_SPEC_ISO_PLANE_CHARSET (coding, 0)) \
2185 *dst++ = c1 & 0x7F; \
2188 else if (charset == CODING_SPEC_ISO_PLANE_CHARSET (coding, 1)) \
2190 *dst++ = c1 | 0x80; \
2194 /* Since CHARSET is not yet invoked to any graphic planes, we \
2195 must invoke it, or, at first, designate it to some graphic \
2196 register. Then repeat the loop to actually produce the \
2198 dst = encode_invocation_designation (charset, coding, dst); \
2201 /* Produce codes for a DIMENSION2 character whose character set is
2202 CHARSET and whose position-codes are C1 and C2. Designation and
2203 invocation codes are also produced in advance if necessary. */
2205 #define ENCODE_ISO_CHARACTER_DIMENSION2(charset, c1, c2) \
2207 if (CODING_SPEC_ISO_SINGLE_SHIFTING (coding)) \
2209 if (coding->flags & CODING_FLAG_ISO_SEVEN_BITS) \
2210 *dst++ = c1 & 0x7F, *dst++ = c2 & 0x7F; \
2212 *dst++ = c1 | 0x80, *dst++ = c2 | 0x80; \
2213 CODING_SPEC_ISO_SINGLE_SHIFTING (coding) = 0; \
2216 else if (charset == CODING_SPEC_ISO_PLANE_CHARSET (coding, 0)) \
2218 *dst++ = c1 & 0x7F, *dst++= c2 & 0x7F; \
2221 else if (charset == CODING_SPEC_ISO_PLANE_CHARSET (coding, 1)) \
2223 *dst++ = c1 | 0x80, *dst++= c2 | 0x80; \
2227 /* Since CHARSET is not yet invoked to any graphic planes, we \
2228 must invoke it, or, at first, designate it to some graphic \
2229 register. Then repeat the loop to actually produce the \
2231 dst = encode_invocation_designation (charset, coding, dst); \
2234 #define ENCODE_ISO_CHARACTER(c) \
2236 int charset, c1, c2; \
2238 SPLIT_CHAR (c, charset, c1, c2); \
2239 if (CHARSET_DEFINED_P (charset)) \
2241 if (CHARSET_DIMENSION (charset) == 1) \
2243 if (charset == CHARSET_ASCII \
2244 && coding->flags & CODING_FLAG_ISO_USE_ROMAN) \
2245 charset = charset_latin_jisx0201; \
2246 ENCODE_ISO_CHARACTER_DIMENSION1 (charset, c1); \
2250 if (charset == charset_jisx0208 \
2251 && coding->flags & CODING_FLAG_ISO_USE_OLDJIS) \
2252 charset = charset_jisx0208_1978; \
2253 ENCODE_ISO_CHARACTER_DIMENSION2 (charset, c1, c2); \
2265 /* Instead of encoding character C, produce one or two `?'s. */
2267 #define ENCODE_UNSAFE_CHARACTER(c) \
2269 ENCODE_ISO_CHARACTER (CODING_INHIBIT_CHARACTER_SUBSTITUTION); \
2270 if (CHARSET_WIDTH (CHAR_CHARSET (c)) > 1) \
2271 ENCODE_ISO_CHARACTER (CODING_INHIBIT_CHARACTER_SUBSTITUTION); \
2275 /* Produce designation and invocation codes at a place pointed by DST
2276 to use CHARSET. The element `spec.iso2022' of *CODING is updated.
2280 encode_invocation_designation (charset
, coding
, dst
)
2282 struct coding_system
*coding
;
2285 int reg
; /* graphic register number */
2287 /* At first, check designations. */
2288 for (reg
= 0; reg
< 4; reg
++)
2289 if (charset
== CODING_SPEC_ISO_DESIGNATION (coding
, reg
))
2294 /* CHARSET is not yet designated to any graphic registers. */
2295 /* At first check the requested designation. */
2296 reg
= CODING_SPEC_ISO_REQUESTED_DESIGNATION (coding
, charset
);
2297 if (reg
== CODING_SPEC_ISO_NO_REQUESTED_DESIGNATION
)
2298 /* Since CHARSET requests no special designation, designate it
2299 to graphic register 0. */
2302 ENCODE_DESIGNATION (charset
, reg
, coding
);
2305 if (CODING_SPEC_ISO_INVOCATION (coding
, 0) != reg
2306 && CODING_SPEC_ISO_INVOCATION (coding
, 1) != reg
)
2308 /* Since the graphic register REG is not invoked to any graphic
2309 planes, invoke it to graphic plane 0. */
2312 case 0: /* graphic register 0 */
2316 case 1: /* graphic register 1 */
2320 case 2: /* graphic register 2 */
2321 if (coding
->flags
& CODING_FLAG_ISO_SINGLE_SHIFT
)
2322 ENCODE_SINGLE_SHIFT_2
;
2324 ENCODE_LOCKING_SHIFT_2
;
2327 case 3: /* graphic register 3 */
2328 if (coding
->flags
& CODING_FLAG_ISO_SINGLE_SHIFT
)
2329 ENCODE_SINGLE_SHIFT_3
;
2331 ENCODE_LOCKING_SHIFT_3
;
2339 /* Produce 2-byte codes for encoded composition rule RULE. */
2341 #define ENCODE_COMPOSITION_RULE(rule) \
2344 COMPOSITION_DECODE_RULE (rule, gref, nref); \
2345 *dst++ = 32 + 81 + gref; \
2346 *dst++ = 32 + nref; \
2349 /* Produce codes for indicating the start of a composition sequence
2350 (ESC 0, ESC 3, or ESC 4). DATA points to an array of integers
2351 which specify information about the composition. See the comment
2352 in coding.h for the format of DATA. */
2354 #define ENCODE_COMPOSITION_START(coding, data) \
2356 coding->composing = data[3]; \
2357 *dst++ = ISO_CODE_ESC; \
2358 if (coding->composing == COMPOSITION_RELATIVE) \
2362 *dst++ = (coding->composing == COMPOSITION_WITH_ALTCHARS \
2364 coding->cmp_data_index = coding->cmp_data_start + 4; \
2365 coding->composition_rule_follows = 0; \
2369 /* Produce codes for indicating the end of the current composition. */
2371 #define ENCODE_COMPOSITION_END(coding, data) \
2373 *dst++ = ISO_CODE_ESC; \
2375 coding->cmp_data_start += data[0]; \
2376 coding->composing = COMPOSITION_NO; \
2377 if (coding->cmp_data_start == coding->cmp_data->used \
2378 && coding->cmp_data->next) \
2380 coding->cmp_data = coding->cmp_data->next; \
2381 coding->cmp_data_start = 0; \
2385 /* Produce composition start sequence ESC 0. Here, this sequence
2386 doesn't mean the start of a new composition but means that we have
2387 just produced components (alternate chars and composition rules) of
2388 the composition and the actual text follows in SRC. */
2390 #define ENCODE_COMPOSITION_FAKE_START(coding) \
2392 *dst++ = ISO_CODE_ESC; \
2394 coding->composing = COMPOSITION_RELATIVE; \
2397 /* The following three macros produce codes for indicating direction
2399 #define ENCODE_CONTROL_SEQUENCE_INTRODUCER \
2401 if (coding->flags == CODING_FLAG_ISO_SEVEN_BITS) \
2402 *dst++ = ISO_CODE_ESC, *dst++ = '['; \
2404 *dst++ = ISO_CODE_CSI; \
2407 #define ENCODE_DIRECTION_R2L \
2408 ENCODE_CONTROL_SEQUENCE_INTRODUCER (dst), *dst++ = '2', *dst++ = ']'
2410 #define ENCODE_DIRECTION_L2R \
2411 ENCODE_CONTROL_SEQUENCE_INTRODUCER (dst), *dst++ = '0', *dst++ = ']'
2413 /* Produce codes for designation and invocation to reset the graphic
2414 planes and registers to initial state. */
2415 #define ENCODE_RESET_PLANE_AND_REGISTER \
2418 if (CODING_SPEC_ISO_INVOCATION (coding, 0) != 0) \
2420 for (reg = 0; reg < 4; reg++) \
2421 if (CODING_SPEC_ISO_INITIAL_DESIGNATION (coding, reg) >= 0 \
2422 && (CODING_SPEC_ISO_DESIGNATION (coding, reg) \
2423 != CODING_SPEC_ISO_INITIAL_DESIGNATION (coding, reg))) \
2424 ENCODE_DESIGNATION \
2425 (CODING_SPEC_ISO_INITIAL_DESIGNATION (coding, reg), reg, coding); \
2428 /* Produce designation sequences of charsets in the line started from
2429 SRC to a place pointed by DST, and return updated DST.
2431 If the current block ends before any end-of-line, we may fail to
2432 find all the necessary designations. */
2434 static unsigned char *
2435 encode_designation_at_bol (coding
, translation_table
, src
, src_end
, dst
)
2436 struct coding_system
*coding
;
2437 Lisp_Object translation_table
;
2438 unsigned char *src
, *src_end
, *dst
;
2440 int charset
, c
, found
= 0, reg
;
2441 /* Table of charsets to be designated to each graphic register. */
2444 for (reg
= 0; reg
< 4; reg
++)
2453 charset
= CHAR_CHARSET (c
);
2454 reg
= CODING_SPEC_ISO_REQUESTED_DESIGNATION (coding
, charset
);
2455 if (reg
!= CODING_SPEC_ISO_NO_REQUESTED_DESIGNATION
&& r
[reg
] < 0)
2465 for (reg
= 0; reg
< 4; reg
++)
2467 && CODING_SPEC_ISO_DESIGNATION (coding
, reg
) != r
[reg
])
2468 ENCODE_DESIGNATION (r
[reg
], reg
, coding
);
2474 /* See the above "GENERAL NOTES on `encode_coding_XXX ()' functions". */
2477 encode_coding_iso2022 (coding
, source
, destination
, src_bytes
, dst_bytes
)
2478 struct coding_system
*coding
;
2479 unsigned char *source
, *destination
;
2480 int src_bytes
, dst_bytes
;
2482 unsigned char *src
= source
;
2483 unsigned char *src_end
= source
+ src_bytes
;
2484 unsigned char *dst
= destination
;
2485 unsigned char *dst_end
= destination
+ dst_bytes
;
2486 /* Since the maximum bytes produced by each loop is 20, we subtract 19
2487 from DST_END to assure overflow checking is necessary only at the
2489 unsigned char *adjusted_dst_end
= dst_end
- 19;
2490 /* SRC_BASE remembers the start position in source in each loop.
2491 The loop will be exited when there's not enough source text to
2492 analyze multi-byte codes (within macro ONE_MORE_CHAR), or when
2493 there's not enough destination area to produce encoded codes
2494 (within macro EMIT_BYTES). */
2495 unsigned char *src_base
;
2497 Lisp_Object translation_table
;
2498 Lisp_Object safe_chars
;
2500 safe_chars
= coding_safe_chars (coding
);
2502 if (NILP (Venable_character_translation
))
2503 translation_table
= Qnil
;
2506 translation_table
= coding
->translation_table_for_encode
;
2507 if (NILP (translation_table
))
2508 translation_table
= Vstandard_translation_table_for_encode
;
2511 coding
->consumed_char
= 0;
2517 if (dst
>= (dst_bytes
? adjusted_dst_end
: (src
- 19)))
2519 coding
->result
= CODING_FINISH_INSUFFICIENT_DST
;
2523 if (coding
->flags
& CODING_FLAG_ISO_DESIGNATE_AT_BOL
2524 && CODING_SPEC_ISO_BOL (coding
))
2526 /* We have to produce designation sequences if any now. */
2527 dst
= encode_designation_at_bol (coding
, translation_table
,
2529 CODING_SPEC_ISO_BOL (coding
) = 0;
2532 /* Check composition start and end. */
2533 if (coding
->composing
!= COMPOSITION_DISABLED
2534 && coding
->cmp_data_start
< coding
->cmp_data
->used
)
2536 struct composition_data
*cmp_data
= coding
->cmp_data
;
2537 int *data
= cmp_data
->data
+ coding
->cmp_data_start
;
2538 int this_pos
= cmp_data
->char_offset
+ coding
->consumed_char
;
2540 if (coding
->composing
== COMPOSITION_RELATIVE
)
2542 if (this_pos
== data
[2])
2544 ENCODE_COMPOSITION_END (coding
, data
);
2545 cmp_data
= coding
->cmp_data
;
2546 data
= cmp_data
->data
+ coding
->cmp_data_start
;
2549 else if (COMPOSING_P (coding
))
2551 /* COMPOSITION_WITH_ALTCHARS or COMPOSITION_WITH_RULE_ALTCHAR */
2552 if (coding
->cmp_data_index
== coding
->cmp_data_start
+ data
[0])
2553 /* We have consumed components of the composition.
2554 What follows in SRC is the composition's base
2556 ENCODE_COMPOSITION_FAKE_START (coding
);
2559 int c
= cmp_data
->data
[coding
->cmp_data_index
++];
2560 if (coding
->composition_rule_follows
)
2562 ENCODE_COMPOSITION_RULE (c
);
2563 coding
->composition_rule_follows
= 0;
2567 if (coding
->flags
& CODING_FLAG_ISO_SAFE
2568 && ! CODING_SAFE_CHAR_P (safe_chars
, c
))
2569 ENCODE_UNSAFE_CHARACTER (c
);
2571 ENCODE_ISO_CHARACTER (c
);
2572 if (coding
->composing
== COMPOSITION_WITH_RULE_ALTCHARS
)
2573 coding
->composition_rule_follows
= 1;
2578 if (!COMPOSING_P (coding
))
2580 if (this_pos
== data
[1])
2582 ENCODE_COMPOSITION_START (coding
, data
);
2590 /* Now encode the character C. */
2591 if (c
< 0x20 || c
== 0x7F)
2595 if (! (coding
->mode
& CODING_MODE_SELECTIVE_DISPLAY
))
2597 if (coding
->flags
& CODING_FLAG_ISO_RESET_AT_CNTL
)
2598 ENCODE_RESET_PLANE_AND_REGISTER
;
2602 /* fall down to treat '\r' as '\n' ... */
2607 if (coding
->flags
& CODING_FLAG_ISO_RESET_AT_EOL
)
2608 ENCODE_RESET_PLANE_AND_REGISTER
;
2609 if (coding
->flags
& CODING_FLAG_ISO_INIT_AT_BOL
)
2610 bcopy (coding
->spec
.iso2022
.initial_designation
,
2611 coding
->spec
.iso2022
.current_designation
,
2612 sizeof coding
->spec
.iso2022
.initial_designation
);
2613 if (coding
->eol_type
== CODING_EOL_LF
2614 || coding
->eol_type
== CODING_EOL_UNDECIDED
)
2615 *dst
++ = ISO_CODE_LF
;
2616 else if (coding
->eol_type
== CODING_EOL_CRLF
)
2617 *dst
++ = ISO_CODE_CR
, *dst
++ = ISO_CODE_LF
;
2619 *dst
++ = ISO_CODE_CR
;
2620 CODING_SPEC_ISO_BOL (coding
) = 1;
2624 if (coding
->flags
& CODING_FLAG_ISO_RESET_AT_CNTL
)
2625 ENCODE_RESET_PLANE_AND_REGISTER
;
2629 else if (ASCII_BYTE_P (c
))
2630 ENCODE_ISO_CHARACTER (c
);
2631 else if (SINGLE_BYTE_CHAR_P (c
))
2636 else if (coding
->flags
& CODING_FLAG_ISO_SAFE
2637 && ! CODING_SAFE_CHAR_P (safe_chars
, c
))
2638 ENCODE_UNSAFE_CHARACTER (c
);
2640 ENCODE_ISO_CHARACTER (c
);
2642 coding
->consumed_char
++;
2646 coding
->consumed
= src_base
- source
;
2647 coding
->produced
= coding
->produced_char
= dst
- destination
;
2651 /*** 4. SJIS and BIG5 handlers ***/
2653 /* Although SJIS and BIG5 are not ISO coding systems, they are used
2654 quite widely. So, for the moment, Emacs supports them in the bare
2655 C code. But, in the future, they may be supported only by CCL. */
2657 /* SJIS is a coding system encoding three character sets: ASCII, right
2658 half of JISX0201-Kana, and JISX0208. An ASCII character is encoded
2659 as is. A character of charset katakana-jisx0201 is encoded by
2660 "position-code + 0x80". A character of charset japanese-jisx0208
2661 is encoded in 2-byte but two position-codes are divided and shifted
2662 so that it fits in the range below.
2664 --- CODE RANGE of SJIS ---
2665 (character set) (range)
2667 KATAKANA-JISX0201 0xA1 .. 0xDF
2668 JISX0208 (1st byte) 0x81 .. 0x9F and 0xE0 .. 0xEF
2669 (2nd byte) 0x40 .. 0x7E and 0x80 .. 0xFC
2670 -------------------------------
2674 /* BIG5 is a coding system encoding two character sets: ASCII and
2675 Big5. An ASCII character is encoded as is. Big5 is a two-byte
2676 character set and is encoded in two bytes.
2678 --- CODE RANGE of BIG5 ---
2679 (character set) (range)
2681 Big5 (1st byte) 0xA1 .. 0xFE
2682 (2nd byte) 0x40 .. 0x7E and 0xA1 .. 0xFE
2683 --------------------------
2685 Since the number of characters in Big5 is larger than maximum
2686 characters in Emacs' charset (96x96), it can't be handled as one
2687 charset. So, in Emacs, Big5 is divided into two: `charset-big5-1'
2688 and `charset-big5-2'. Both are DIMENSION2 and CHARS94. The former
2689 contains frequently used characters and the latter contains less
2690 frequently used characters. */
2692 /* Macros to decode or encode a character of Big5 in BIG5. B1 and B2
2693 are the 1st and 2nd position-codes of Big5 in BIG5 coding system.
2694 C1 and C2 are the 1st and 2nd position-codes of Emacs' internal
2695 format. CHARSET is `charset_big5_1' or `charset_big5_2'. */
2697 /* Number of Big5 characters which have the same code in 1st byte. */
2698 #define BIG5_SAME_ROW (0xFF - 0xA1 + 0x7F - 0x40)
2700 #define DECODE_BIG5(b1, b2, charset, c1, c2) \
2703 = (b1 - 0xA1) * BIG5_SAME_ROW + b2 - (b2 < 0x7F ? 0x40 : 0x62); \
2705 charset = charset_big5_1; \
2708 charset = charset_big5_2; \
2709 temp -= (0xC9 - 0xA1) * BIG5_SAME_ROW; \
2711 c1 = temp / (0xFF - 0xA1) + 0x21; \
2712 c2 = temp % (0xFF - 0xA1) + 0x21; \
2715 #define ENCODE_BIG5(charset, c1, c2, b1, b2) \
2717 unsigned int temp = (c1 - 0x21) * (0xFF - 0xA1) + (c2 - 0x21); \
2718 if (charset == charset_big5_2) \
2719 temp += BIG5_SAME_ROW * (0xC9 - 0xA1); \
2720 b1 = temp / BIG5_SAME_ROW + 0xA1; \
2721 b2 = temp % BIG5_SAME_ROW; \
2722 b2 += b2 < 0x3F ? 0x40 : 0x62; \
2725 /* See the above "GENERAL NOTES on `detect_coding_XXX ()' functions".
2726 Check if a text is encoded in SJIS. If it is, return
2727 CODING_CATEGORY_MASK_SJIS, else return 0. */
2730 detect_coding_sjis (src
, src_end
, multibytep
)
2731 unsigned char *src
, *src_end
;
2735 /* Dummy for ONE_MORE_BYTE. */
2736 struct coding_system dummy_coding
;
2737 struct coding_system
*coding
= &dummy_coding
;
2741 ONE_MORE_BYTE_CHECK_MULTIBYTE (c
, multibytep
);
2744 if (c
== 0x80 || c
== 0xA0 || c
> 0xEF)
2746 if (c
<= 0x9F || c
>= 0xE0)
2748 ONE_MORE_BYTE_CHECK_MULTIBYTE (c
, multibytep
);
2749 if (c
< 0x40 || c
== 0x7F || c
> 0xFC)
2754 return CODING_CATEGORY_MASK_SJIS
;
2757 /* See the above "GENERAL NOTES on `detect_coding_XXX ()' functions".
2758 Check if a text is encoded in BIG5. If it is, return
2759 CODING_CATEGORY_MASK_BIG5, else return 0. */
2762 detect_coding_big5 (src
, src_end
, multibytep
)
2763 unsigned char *src
, *src_end
;
2767 /* Dummy for ONE_MORE_BYTE. */
2768 struct coding_system dummy_coding
;
2769 struct coding_system
*coding
= &dummy_coding
;
2773 ONE_MORE_BYTE_CHECK_MULTIBYTE (c
, multibytep
);
2776 if (c
< 0xA1 || c
> 0xFE)
2778 ONE_MORE_BYTE_CHECK_MULTIBYTE (c
, multibytep
);
2779 if (c
< 0x40 || (c
> 0x7F && c
< 0xA1) || c
> 0xFE)
2783 return CODING_CATEGORY_MASK_BIG5
;
2786 /* See the above "GENERAL NOTES on `detect_coding_XXX ()' functions".
2787 Check if a text is encoded in UTF-8. If it is, return
2788 CODING_CATEGORY_MASK_UTF_8, else return 0. */
2790 #define UTF_8_1_OCTET_P(c) ((c) < 0x80)
2791 #define UTF_8_EXTRA_OCTET_P(c) (((c) & 0xC0) == 0x80)
2792 #define UTF_8_2_OCTET_LEADING_P(c) (((c) & 0xE0) == 0xC0)
2793 #define UTF_8_3_OCTET_LEADING_P(c) (((c) & 0xF0) == 0xE0)
2794 #define UTF_8_4_OCTET_LEADING_P(c) (((c) & 0xF8) == 0xF0)
2795 #define UTF_8_5_OCTET_LEADING_P(c) (((c) & 0xFC) == 0xF8)
2796 #define UTF_8_6_OCTET_LEADING_P(c) (((c) & 0xFE) == 0xFC)
2799 detect_coding_utf_8 (src
, src_end
, multibytep
)
2800 unsigned char *src
, *src_end
;
2804 int seq_maybe_bytes
;
2805 /* Dummy for ONE_MORE_BYTE. */
2806 struct coding_system dummy_coding
;
2807 struct coding_system
*coding
= &dummy_coding
;
2811 ONE_MORE_BYTE_CHECK_MULTIBYTE (c
, multibytep
);
2812 if (UTF_8_1_OCTET_P (c
))
2814 else if (UTF_8_2_OCTET_LEADING_P (c
))
2815 seq_maybe_bytes
= 1;
2816 else if (UTF_8_3_OCTET_LEADING_P (c
))
2817 seq_maybe_bytes
= 2;
2818 else if (UTF_8_4_OCTET_LEADING_P (c
))
2819 seq_maybe_bytes
= 3;
2820 else if (UTF_8_5_OCTET_LEADING_P (c
))
2821 seq_maybe_bytes
= 4;
2822 else if (UTF_8_6_OCTET_LEADING_P (c
))
2823 seq_maybe_bytes
= 5;
2829 ONE_MORE_BYTE_CHECK_MULTIBYTE (c
, multibytep
);
2830 if (!UTF_8_EXTRA_OCTET_P (c
))
2834 while (seq_maybe_bytes
> 0);
2838 return CODING_CATEGORY_MASK_UTF_8
;
2841 /* See the above "GENERAL NOTES on `detect_coding_XXX ()' functions".
2842 Check if a text is encoded in UTF-16 Big Endian (endian == 1) or
2843 Little Endian (otherwise). If it is, return
2844 CODING_CATEGORY_MASK_UTF_16_BE or CODING_CATEGORY_MASK_UTF_16_LE,
2847 #define UTF_16_INVALID_P(val) \
2848 (((val) == 0xFFFE) \
2849 || ((val) == 0xFFFF))
2851 #define UTF_16_HIGH_SURROGATE_P(val) \
2852 (((val) & 0xD800) == 0xD800)
2854 #define UTF_16_LOW_SURROGATE_P(val) \
2855 (((val) & 0xDC00) == 0xDC00)
2858 detect_coding_utf_16 (src
, src_end
, multibytep
)
2859 unsigned char *src
, *src_end
;
2862 unsigned char c1
, c2
;
2863 /* Dummy for TWO_MORE_BYTES. */
2864 struct coding_system dummy_coding
;
2865 struct coding_system
*coding
= &dummy_coding
;
2867 ONE_MORE_BYTE_CHECK_MULTIBYTE (c1
, multibytep
);
2868 ONE_MORE_BYTE_CHECK_MULTIBYTE (c2
, multibytep
);
2870 if ((c1
== 0xFF) && (c2
== 0xFE))
2871 return CODING_CATEGORY_MASK_UTF_16_LE
;
2872 else if ((c1
== 0xFE) && (c2
== 0xFF))
2873 return CODING_CATEGORY_MASK_UTF_16_BE
;
2879 /* See the above "GENERAL NOTES on `decode_coding_XXX ()' functions".
2880 If SJIS_P is 1, decode SJIS text, else decode BIG5 test. */
2883 decode_coding_sjis_big5 (coding
, source
, destination
,
2884 src_bytes
, dst_bytes
, sjis_p
)
2885 struct coding_system
*coding
;
2886 unsigned char *source
, *destination
;
2887 int src_bytes
, dst_bytes
;
2890 unsigned char *src
= source
;
2891 unsigned char *src_end
= source
+ src_bytes
;
2892 unsigned char *dst
= destination
;
2893 unsigned char *dst_end
= destination
+ dst_bytes
;
2894 /* SRC_BASE remembers the start position in source in each loop.
2895 The loop will be exited when there's not enough source code
2896 (within macro ONE_MORE_BYTE), or when there's not enough
2897 destination area to produce a character (within macro
2899 unsigned char *src_base
;
2900 Lisp_Object translation_table
;
2902 if (NILP (Venable_character_translation
))
2903 translation_table
= Qnil
;
2906 translation_table
= coding
->translation_table_for_decode
;
2907 if (NILP (translation_table
))
2908 translation_table
= Vstandard_translation_table_for_decode
;
2911 coding
->produced_char
= 0;
2914 int c
, charset
, c1
, c2
;
2921 charset
= CHARSET_ASCII
;
2926 if (coding
->eol_type
== CODING_EOL_CRLF
)
2931 else if (coding
->mode
2932 & CODING_MODE_INHIBIT_INCONSISTENT_EOL
)
2934 coding
->result
= CODING_FINISH_INCONSISTENT_EOL
;
2935 goto label_end_of_loop
;
2938 /* To process C2 again, SRC is subtracted by 1. */
2941 else if (coding
->eol_type
== CODING_EOL_CR
)
2945 && (coding
->mode
& CODING_MODE_INHIBIT_INCONSISTENT_EOL
)
2946 && (coding
->eol_type
== CODING_EOL_CR
2947 || coding
->eol_type
== CODING_EOL_CRLF
))
2949 coding
->result
= CODING_FINISH_INCONSISTENT_EOL
;
2950 goto label_end_of_loop
;
2958 if (c1
== 0x80 || c1
== 0xA0 || c1
> 0xEF)
2959 goto label_invalid_code
;
2960 if (c1
<= 0x9F || c1
>= 0xE0)
2962 /* SJIS -> JISX0208 */
2964 if (c2
< 0x40 || c2
== 0x7F || c2
> 0xFC)
2965 goto label_invalid_code
;
2966 DECODE_SJIS (c1
, c2
, c1
, c2
);
2967 charset
= charset_jisx0208
;
2970 /* SJIS -> JISX0201-Kana */
2971 charset
= charset_katakana_jisx0201
;
2976 if (c1
< 0xA0 || c1
> 0xFE)
2977 goto label_invalid_code
;
2979 if (c2
< 0x40 || (c2
> 0x7E && c2
< 0xA1) || c2
> 0xFE)
2980 goto label_invalid_code
;
2981 DECODE_BIG5 (c1
, c2
, charset
, c1
, c2
);
2985 c
= DECODE_ISO_CHARACTER (charset
, c1
, c2
);
2997 coding
->consumed
= coding
->consumed_char
= src_base
- source
;
2998 coding
->produced
= dst
- destination
;
3002 /* See the above "GENERAL NOTES on `encode_coding_XXX ()' functions".
3003 This function can encode charsets `ascii', `katakana-jisx0201',
3004 `japanese-jisx0208', `chinese-big5-1', and `chinese-big5-2'. We
3005 are sure that all these charsets are registered as official charset
3006 (i.e. do not have extended leading-codes). Characters of other
3007 charsets are produced without any encoding. If SJIS_P is 1, encode
3008 SJIS text, else encode BIG5 text. */
3011 encode_coding_sjis_big5 (coding
, source
, destination
,
3012 src_bytes
, dst_bytes
, sjis_p
)
3013 struct coding_system
*coding
;
3014 unsigned char *source
, *destination
;
3015 int src_bytes
, dst_bytes
;
3018 unsigned char *src
= source
;
3019 unsigned char *src_end
= source
+ src_bytes
;
3020 unsigned char *dst
= destination
;
3021 unsigned char *dst_end
= destination
+ dst_bytes
;
3022 /* SRC_BASE remembers the start position in source in each loop.
3023 The loop will be exited when there's not enough source text to
3024 analyze multi-byte codes (within macro ONE_MORE_CHAR), or when
3025 there's not enough destination area to produce encoded codes
3026 (within macro EMIT_BYTES). */
3027 unsigned char *src_base
;
3028 Lisp_Object translation_table
;
3030 if (NILP (Venable_character_translation
))
3031 translation_table
= Qnil
;
3034 translation_table
= coding
->translation_table_for_encode
;
3035 if (NILP (translation_table
))
3036 translation_table
= Vstandard_translation_table_for_encode
;
3041 int c
, charset
, c1
, c2
;
3046 /* Now encode the character C. */
3047 if (SINGLE_BYTE_CHAR_P (c
))
3052 if (!(coding
->mode
& CODING_MODE_SELECTIVE_DISPLAY
))
3059 if (coding
->eol_type
== CODING_EOL_CRLF
)
3061 EMIT_TWO_BYTES ('\r', c
);
3064 else if (coding
->eol_type
== CODING_EOL_CR
)
3072 SPLIT_CHAR (c
, charset
, c1
, c2
);
3075 if (charset
== charset_jisx0208
3076 || charset
== charset_jisx0208_1978
)
3078 ENCODE_SJIS (c1
, c2
, c1
, c2
);
3079 EMIT_TWO_BYTES (c1
, c2
);
3081 else if (charset
== charset_katakana_jisx0201
)
3082 EMIT_ONE_BYTE (c1
| 0x80);
3083 else if (charset
== charset_latin_jisx0201
)
3086 /* There's no way other than producing the internal
3088 EMIT_BYTES (src_base
, src
);
3092 if (charset
== charset_big5_1
|| charset
== charset_big5_2
)
3094 ENCODE_BIG5 (charset
, c1
, c2
, c1
, c2
);
3095 EMIT_TWO_BYTES (c1
, c2
);
3098 /* There's no way other than producing the internal
3100 EMIT_BYTES (src_base
, src
);
3103 coding
->consumed_char
++;
3107 coding
->consumed
= src_base
- source
;
3108 coding
->produced
= coding
->produced_char
= dst
- destination
;
3112 /*** 5. CCL handlers ***/
3114 /* See the above "GENERAL NOTES on `detect_coding_XXX ()' functions".
3115 Check if a text is encoded in a coding system of which
3116 encoder/decoder are written in CCL program. If it is, return
3117 CODING_CATEGORY_MASK_CCL, else return 0. */
3120 detect_coding_ccl (src
, src_end
, multibytep
)
3121 unsigned char *src
, *src_end
;
3124 unsigned char *valid
;
3126 /* Dummy for ONE_MORE_BYTE. */
3127 struct coding_system dummy_coding
;
3128 struct coding_system
*coding
= &dummy_coding
;
3130 /* No coding system is assigned to coding-category-ccl. */
3131 if (!coding_system_table
[CODING_CATEGORY_IDX_CCL
])
3134 valid
= coding_system_table
[CODING_CATEGORY_IDX_CCL
]->spec
.ccl
.valid_codes
;
3137 ONE_MORE_BYTE_CHECK_MULTIBYTE (c
, multibytep
);
3142 return CODING_CATEGORY_MASK_CCL
;
3146 /*** 6. End-of-line handlers ***/
3148 /* See the above "GENERAL NOTES on `decode_coding_XXX ()' functions". */
3151 decode_eol (coding
, source
, destination
, src_bytes
, dst_bytes
)
3152 struct coding_system
*coding
;
3153 unsigned char *source
, *destination
;
3154 int src_bytes
, dst_bytes
;
3156 unsigned char *src
= source
;
3157 unsigned char *dst
= destination
;
3158 unsigned char *src_end
= src
+ src_bytes
;
3159 unsigned char *dst_end
= dst
+ dst_bytes
;
3160 Lisp_Object translation_table
;
3161 /* SRC_BASE remembers the start position in source in each loop.
3162 The loop will be exited when there's not enough source code
3163 (within macro ONE_MORE_BYTE), or when there's not enough
3164 destination area to produce a character (within macro
3166 unsigned char *src_base
;
3169 translation_table
= Qnil
;
3170 switch (coding
->eol_type
)
3172 case CODING_EOL_CRLF
:
3182 if (coding
->mode
& CODING_MODE_INHIBIT_INCONSISTENT_EOL
)
3184 coding
->result
= CODING_FINISH_INCONSISTENT_EOL
;
3185 goto label_end_of_loop
;
3192 && (coding
->mode
& CODING_MODE_INHIBIT_INCONSISTENT_EOL
))
3194 coding
->result
= CODING_FINISH_INCONSISTENT_EOL
;
3195 goto label_end_of_loop
;
3208 if (coding
->mode
& CODING_MODE_INHIBIT_INCONSISTENT_EOL
)
3210 coding
->result
= CODING_FINISH_INCONSISTENT_EOL
;
3211 goto label_end_of_loop
;
3220 default: /* no need for EOL handling */
3230 coding
->consumed
= coding
->consumed_char
= src_base
- source
;
3231 coding
->produced
= dst
- destination
;
3235 /* See "GENERAL NOTES about `encode_coding_XXX ()' functions". Encode
3236 format of end-of-line according to `coding->eol_type'. It also
3237 convert multibyte form 8-bit characters to unibyte if
3238 CODING->src_multibyte is nonzero. If `coding->mode &
3239 CODING_MODE_SELECTIVE_DISPLAY' is nonzero, code '\r' in source text
3240 also means end-of-line. */
3243 encode_eol (coding
, source
, destination
, src_bytes
, dst_bytes
)
3244 struct coding_system
*coding
;
3245 const unsigned char *source
;
3246 unsigned char *destination
;
3247 int src_bytes
, dst_bytes
;
3249 const unsigned char *src
= source
;
3250 unsigned char *dst
= destination
;
3251 const unsigned char *src_end
= src
+ src_bytes
;
3252 unsigned char *dst_end
= dst
+ dst_bytes
;
3253 Lisp_Object translation_table
;
3254 /* SRC_BASE remembers the start position in source in each loop.
3255 The loop will be exited when there's not enough source text to
3256 analyze multi-byte codes (within macro ONE_MORE_CHAR), or when
3257 there's not enough destination area to produce encoded codes
3258 (within macro EMIT_BYTES). */
3259 const unsigned char *src_base
;
3262 int selective_display
= coding
->mode
& CODING_MODE_SELECTIVE_DISPLAY
;
3264 translation_table
= Qnil
;
3265 if (coding
->src_multibyte
3266 && *(src_end
- 1) == LEADING_CODE_8_BIT_CONTROL
)
3270 coding
->result
= CODING_FINISH_INSUFFICIENT_SRC
;
3273 if (coding
->eol_type
== CODING_EOL_CRLF
)
3275 while (src
< src_end
)
3281 else if (c
== '\n' || (c
== '\r' && selective_display
))
3282 EMIT_TWO_BYTES ('\r', '\n');
3292 if (!dst_bytes
|| src_bytes
<= dst_bytes
)
3294 safe_bcopy (src
, dst
, src_bytes
);
3300 if (coding
->src_multibyte
3301 && *(src
+ dst_bytes
- 1) == LEADING_CODE_8_BIT_CONTROL
)
3303 safe_bcopy (src
, dst
, dst_bytes
);
3304 src_base
= src
+ dst_bytes
;
3305 dst
= destination
+ dst_bytes
;
3306 coding
->result
= CODING_FINISH_INSUFFICIENT_DST
;
3308 if (coding
->eol_type
== CODING_EOL_CR
)
3310 for (tmp
= destination
; tmp
< dst
; tmp
++)
3311 if (*tmp
== '\n') *tmp
= '\r';
3313 else if (selective_display
)
3315 for (tmp
= destination
; tmp
< dst
; tmp
++)
3316 if (*tmp
== '\r') *tmp
= '\n';
3319 if (coding
->src_multibyte
)
3320 dst
= destination
+ str_as_unibyte (destination
, dst
- destination
);
3322 coding
->consumed
= src_base
- source
;
3323 coding
->produced
= dst
- destination
;
3324 coding
->produced_char
= coding
->produced
;
3328 /*** 7. C library functions ***/
3330 /* In Emacs Lisp, a coding system is represented by a Lisp symbol which
3331 has a property `coding-system'. The value of this property is a
3332 vector of length 5 (called the coding-vector). Among elements of
3333 this vector, the first (element[0]) and the fifth (element[4])
3334 carry important information for decoding/encoding. Before
3335 decoding/encoding, this information should be set in fields of a
3336 structure of type `coding_system'.
3338 The value of the property `coding-system' can be a symbol of another
3339 subsidiary coding-system. In that case, Emacs gets coding-vector
3342 `element[0]' contains information to be set in `coding->type'. The
3343 value and its meaning is as follows:
3345 0 -- coding_type_emacs_mule
3346 1 -- coding_type_sjis
3347 2 -- coding_type_iso2022
3348 3 -- coding_type_big5
3349 4 -- coding_type_ccl encoder/decoder written in CCL
3350 nil -- coding_type_no_conversion
3351 t -- coding_type_undecided (automatic conversion on decoding,
3352 no-conversion on encoding)
3354 `element[4]' contains information to be set in `coding->flags' and
3355 `coding->spec'. The meaning varies by `coding->type'.
3357 If `coding->type' is `coding_type_iso2022', element[4] is a vector
3358 of length 32 (of which the first 13 sub-elements are used now).
3359 Meanings of these sub-elements are:
3361 sub-element[N] where N is 0 through 3: to be set in `coding->spec.iso2022'
3362 If the value is an integer of valid charset, the charset is
3363 assumed to be designated to graphic register N initially.
3365 If the value is minus, it is a minus value of charset which
3366 reserves graphic register N, which means that the charset is
3367 not designated initially but should be designated to graphic
3368 register N just before encoding a character in that charset.
3370 If the value is nil, graphic register N is never used on
3373 sub-element[N] where N is 4 through 11: to be set in `coding->flags'
3374 Each value takes t or nil. See the section ISO2022 of
3375 `coding.h' for more information.
3377 If `coding->type' is `coding_type_big5', element[4] is t to denote
3378 BIG5-ETen or nil to denote BIG5-HKU.
3380 If `coding->type' takes the other value, element[4] is ignored.
3382 Emacs Lisp's coding systems also carry information about format of
3383 end-of-line in a value of property `eol-type'. If the value is
3384 integer, 0 means CODING_EOL_LF, 1 means CODING_EOL_CRLF, and 2
3385 means CODING_EOL_CR. If it is not integer, it should be a vector
3386 of subsidiary coding systems of which property `eol-type' has one
3387 of the above values.
3391 /* Extract information for decoding/encoding from CODING_SYSTEM_SYMBOL
3392 and set it in CODING. If CODING_SYSTEM_SYMBOL is invalid, CODING
3393 is setup so that no conversion is necessary and return -1, else
3397 setup_coding_system (coding_system
, coding
)
3398 Lisp_Object coding_system
;
3399 struct coding_system
*coding
;
3401 Lisp_Object coding_spec
, coding_type
, eol_type
, plist
;
3404 /* At first, zero clear all members. */
3405 bzero (coding
, sizeof (struct coding_system
));
3407 /* Initialize some fields required for all kinds of coding systems. */
3408 coding
->symbol
= coding_system
;
3409 coding
->heading_ascii
= -1;
3410 coding
->post_read_conversion
= coding
->pre_write_conversion
= Qnil
;
3411 coding
->composing
= COMPOSITION_DISABLED
;
3412 coding
->cmp_data
= NULL
;
3414 if (NILP (coding_system
))
3415 goto label_invalid_coding_system
;
3417 coding_spec
= Fget (coding_system
, Qcoding_system
);
3419 if (!VECTORP (coding_spec
)
3420 || XVECTOR (coding_spec
)->size
!= 5
3421 || !CONSP (XVECTOR (coding_spec
)->contents
[3]))
3422 goto label_invalid_coding_system
;
3424 eol_type
= inhibit_eol_conversion
? Qnil
: Fget (coding_system
, Qeol_type
);
3425 if (VECTORP (eol_type
))
3427 coding
->eol_type
= CODING_EOL_UNDECIDED
;
3428 coding
->common_flags
= CODING_REQUIRE_DETECTION_MASK
;
3430 else if (XFASTINT (eol_type
) == 1)
3432 coding
->eol_type
= CODING_EOL_CRLF
;
3433 coding
->common_flags
3434 = CODING_REQUIRE_DECODING_MASK
| CODING_REQUIRE_ENCODING_MASK
;
3436 else if (XFASTINT (eol_type
) == 2)
3438 coding
->eol_type
= CODING_EOL_CR
;
3439 coding
->common_flags
3440 = CODING_REQUIRE_DECODING_MASK
| CODING_REQUIRE_ENCODING_MASK
;
3443 coding
->eol_type
= CODING_EOL_LF
;
3445 coding_type
= XVECTOR (coding_spec
)->contents
[0];
3446 /* Try short cut. */
3447 if (SYMBOLP (coding_type
))
3449 if (EQ (coding_type
, Qt
))
3451 coding
->type
= coding_type_undecided
;
3452 coding
->common_flags
|= CODING_REQUIRE_DETECTION_MASK
;
3455 coding
->type
= coding_type_no_conversion
;
3456 /* Initialize this member. Any thing other than
3457 CODING_CATEGORY_IDX_UTF_16_BE and
3458 CODING_CATEGORY_IDX_UTF_16_LE are ok because they have
3459 special treatment in detect_eol. */
3460 coding
->category_idx
= CODING_CATEGORY_IDX_EMACS_MULE
;
3465 /* Get values of coding system properties:
3466 `post-read-conversion', `pre-write-conversion',
3467 `translation-table-for-decode', `translation-table-for-encode'. */
3468 plist
= XVECTOR (coding_spec
)->contents
[3];
3469 /* Pre & post conversion functions should be disabled if
3470 inhibit_eol_conversion is nonzero. This is the case that a code
3471 conversion function is called while those functions are running. */
3472 if (! inhibit_pre_post_conversion
)
3474 coding
->post_read_conversion
= Fplist_get (plist
, Qpost_read_conversion
);
3475 coding
->pre_write_conversion
= Fplist_get (plist
, Qpre_write_conversion
);
3477 val
= Fplist_get (plist
, Qtranslation_table_for_decode
);
3479 val
= Fget (val
, Qtranslation_table_for_decode
);
3480 coding
->translation_table_for_decode
= CHAR_TABLE_P (val
) ? val
: Qnil
;
3481 val
= Fplist_get (plist
, Qtranslation_table_for_encode
);
3483 val
= Fget (val
, Qtranslation_table_for_encode
);
3484 coding
->translation_table_for_encode
= CHAR_TABLE_P (val
) ? val
: Qnil
;
3485 val
= Fplist_get (plist
, Qcoding_category
);
3488 val
= Fget (val
, Qcoding_category_index
);
3490 coding
->category_idx
= XINT (val
);
3492 goto label_invalid_coding_system
;
3495 goto label_invalid_coding_system
;
3497 /* If the coding system has non-nil `composition' property, enable
3498 composition handling. */
3499 val
= Fplist_get (plist
, Qcomposition
);
3501 coding
->composing
= COMPOSITION_NO
;
3503 switch (XFASTINT (coding_type
))
3506 coding
->type
= coding_type_emacs_mule
;
3507 coding
->common_flags
3508 |= CODING_REQUIRE_DECODING_MASK
| CODING_REQUIRE_ENCODING_MASK
;
3509 coding
->composing
= COMPOSITION_NO
;
3510 if (!NILP (coding
->post_read_conversion
))
3511 coding
->common_flags
|= CODING_REQUIRE_DECODING_MASK
;
3512 if (!NILP (coding
->pre_write_conversion
))
3513 coding
->common_flags
|= CODING_REQUIRE_ENCODING_MASK
;
3517 coding
->type
= coding_type_sjis
;
3518 coding
->common_flags
3519 |= CODING_REQUIRE_DECODING_MASK
| CODING_REQUIRE_ENCODING_MASK
;
3523 coding
->type
= coding_type_iso2022
;
3524 coding
->common_flags
3525 |= CODING_REQUIRE_DECODING_MASK
| CODING_REQUIRE_ENCODING_MASK
;
3527 Lisp_Object val
, temp
;
3529 int i
, charset
, reg_bits
= 0;
3531 val
= XVECTOR (coding_spec
)->contents
[4];
3533 if (!VECTORP (val
) || XVECTOR (val
)->size
!= 32)
3534 goto label_invalid_coding_system
;
3536 flags
= XVECTOR (val
)->contents
;
3538 = ((NILP (flags
[4]) ? 0 : CODING_FLAG_ISO_SHORT_FORM
)
3539 | (NILP (flags
[5]) ? 0 : CODING_FLAG_ISO_RESET_AT_EOL
)
3540 | (NILP (flags
[6]) ? 0 : CODING_FLAG_ISO_RESET_AT_CNTL
)
3541 | (NILP (flags
[7]) ? 0 : CODING_FLAG_ISO_SEVEN_BITS
)
3542 | (NILP (flags
[8]) ? 0 : CODING_FLAG_ISO_LOCKING_SHIFT
)
3543 | (NILP (flags
[9]) ? 0 : CODING_FLAG_ISO_SINGLE_SHIFT
)
3544 | (NILP (flags
[10]) ? 0 : CODING_FLAG_ISO_USE_ROMAN
)
3545 | (NILP (flags
[11]) ? 0 : CODING_FLAG_ISO_USE_OLDJIS
)
3546 | (NILP (flags
[12]) ? 0 : CODING_FLAG_ISO_NO_DIRECTION
)
3547 | (NILP (flags
[13]) ? 0 : CODING_FLAG_ISO_INIT_AT_BOL
)
3548 | (NILP (flags
[14]) ? 0 : CODING_FLAG_ISO_DESIGNATE_AT_BOL
)
3549 | (NILP (flags
[15]) ? 0 : CODING_FLAG_ISO_SAFE
)
3550 | (NILP (flags
[16]) ? 0 : CODING_FLAG_ISO_LATIN_EXTRA
)
3553 /* Invoke graphic register 0 to plane 0. */
3554 CODING_SPEC_ISO_INVOCATION (coding
, 0) = 0;
3555 /* Invoke graphic register 1 to plane 1 if we can use full 8-bit. */
3556 CODING_SPEC_ISO_INVOCATION (coding
, 1)
3557 = (coding
->flags
& CODING_FLAG_ISO_SEVEN_BITS
? -1 : 1);
3558 /* Not single shifting at first. */
3559 CODING_SPEC_ISO_SINGLE_SHIFTING (coding
) = 0;
3560 /* Beginning of buffer should also be regarded as bol. */
3561 CODING_SPEC_ISO_BOL (coding
) = 1;
3563 for (charset
= 0; charset
<= MAX_CHARSET
; charset
++)
3564 CODING_SPEC_ISO_REVISION_NUMBER (coding
, charset
) = 255;
3565 val
= Vcharset_revision_alist
;
3568 charset
= get_charset_id (Fcar_safe (XCAR (val
)));
3570 && (temp
= Fcdr_safe (XCAR (val
)), INTEGERP (temp
))
3571 && (i
= XINT (temp
), (i
>= 0 && (i
+ '@') < 128)))
3572 CODING_SPEC_ISO_REVISION_NUMBER (coding
, charset
) = i
;
3576 /* Checks FLAGS[REG] (REG = 0, 1, 2 3) and decide designations.
3577 FLAGS[REG] can be one of below:
3578 integer CHARSET: CHARSET occupies register I,
3579 t: designate nothing to REG initially, but can be used
3581 list of integer, nil, or t: designate the first
3582 element (if integer) to REG initially, the remaining
3583 elements (if integer) is designated to REG on request,
3584 if an element is t, REG can be used by any charsets,
3585 nil: REG is never used. */
3586 for (charset
= 0; charset
<= MAX_CHARSET
; charset
++)
3587 CODING_SPEC_ISO_REQUESTED_DESIGNATION (coding
, charset
)
3588 = CODING_SPEC_ISO_NO_REQUESTED_DESIGNATION
;
3589 for (i
= 0; i
< 4; i
++)
3591 if ((INTEGERP (flags
[i
])
3592 && (charset
= XINT (flags
[i
]), CHARSET_VALID_P (charset
)))
3593 || (charset
= get_charset_id (flags
[i
])) >= 0)
3595 CODING_SPEC_ISO_INITIAL_DESIGNATION (coding
, i
) = charset
;
3596 CODING_SPEC_ISO_REQUESTED_DESIGNATION (coding
, charset
) = i
;
3598 else if (EQ (flags
[i
], Qt
))
3600 CODING_SPEC_ISO_INITIAL_DESIGNATION (coding
, i
) = -1;
3602 coding
->flags
|= CODING_FLAG_ISO_DESIGNATION
;
3604 else if (CONSP (flags
[i
]))
3609 coding
->flags
|= CODING_FLAG_ISO_DESIGNATION
;
3610 if ((INTEGERP (XCAR (tail
))
3611 && (charset
= XINT (XCAR (tail
)),
3612 CHARSET_VALID_P (charset
)))
3613 || (charset
= get_charset_id (XCAR (tail
))) >= 0)
3615 CODING_SPEC_ISO_INITIAL_DESIGNATION (coding
, i
) = charset
;
3616 CODING_SPEC_ISO_REQUESTED_DESIGNATION (coding
, charset
) =i
;
3619 CODING_SPEC_ISO_INITIAL_DESIGNATION (coding
, i
) = -1;
3621 while (CONSP (tail
))
3623 if ((INTEGERP (XCAR (tail
))
3624 && (charset
= XINT (XCAR (tail
)),
3625 CHARSET_VALID_P (charset
)))
3626 || (charset
= get_charset_id (XCAR (tail
))) >= 0)
3627 CODING_SPEC_ISO_REQUESTED_DESIGNATION (coding
, charset
)
3629 else if (EQ (XCAR (tail
), Qt
))
3635 CODING_SPEC_ISO_INITIAL_DESIGNATION (coding
, i
) = -1;
3637 CODING_SPEC_ISO_DESIGNATION (coding
, i
)
3638 = CODING_SPEC_ISO_INITIAL_DESIGNATION (coding
, i
);
3641 if (reg_bits
&& ! (coding
->flags
& CODING_FLAG_ISO_LOCKING_SHIFT
))
3643 /* REG 1 can be used only by locking shift in 7-bit env. */
3644 if (coding
->flags
& CODING_FLAG_ISO_SEVEN_BITS
)
3646 if (! (coding
->flags
& CODING_FLAG_ISO_SINGLE_SHIFT
))
3647 /* Without any shifting, only REG 0 and 1 can be used. */
3652 for (charset
= 0; charset
<= MAX_CHARSET
; charset
++)
3654 if (CHARSET_DEFINED_P (charset
)
3655 && (CODING_SPEC_ISO_REQUESTED_DESIGNATION (coding
, charset
)
3656 == CODING_SPEC_ISO_NO_REQUESTED_DESIGNATION
))
3658 /* There exist some default graphic registers to be
3661 /* We had better avoid designating a charset of
3662 CHARS96 to REG 0 as far as possible. */
3663 if (CHARSET_CHARS (charset
) == 96)
3664 CODING_SPEC_ISO_REQUESTED_DESIGNATION (coding
, charset
)
3666 ? 1 : (reg_bits
& 4 ? 2 : (reg_bits
& 8 ? 3 : 0)));
3668 CODING_SPEC_ISO_REQUESTED_DESIGNATION (coding
, charset
)
3670 ? 0 : (reg_bits
& 2 ? 1 : (reg_bits
& 4 ? 2 : 3)));
3674 coding
->common_flags
|= CODING_REQUIRE_FLUSHING_MASK
;
3675 coding
->spec
.iso2022
.last_invalid_designation_register
= -1;
3679 coding
->type
= coding_type_big5
;
3680 coding
->common_flags
3681 |= CODING_REQUIRE_DECODING_MASK
| CODING_REQUIRE_ENCODING_MASK
;
3683 = (NILP (XVECTOR (coding_spec
)->contents
[4])
3684 ? CODING_FLAG_BIG5_HKU
3685 : CODING_FLAG_BIG5_ETEN
);
3689 coding
->type
= coding_type_ccl
;
3690 coding
->common_flags
3691 |= CODING_REQUIRE_DECODING_MASK
| CODING_REQUIRE_ENCODING_MASK
;
3693 val
= XVECTOR (coding_spec
)->contents
[4];
3695 || setup_ccl_program (&(coding
->spec
.ccl
.decoder
),
3697 || setup_ccl_program (&(coding
->spec
.ccl
.encoder
),
3699 goto label_invalid_coding_system
;
3701 bzero (coding
->spec
.ccl
.valid_codes
, 256);
3702 val
= Fplist_get (plist
, Qvalid_codes
);
3707 for (; CONSP (val
); val
= XCDR (val
))
3711 && XINT (this) >= 0 && XINT (this) < 256)
3712 coding
->spec
.ccl
.valid_codes
[XINT (this)] = 1;
3713 else if (CONSP (this)
3714 && INTEGERP (XCAR (this))
3715 && INTEGERP (XCDR (this)))
3717 int start
= XINT (XCAR (this));
3718 int end
= XINT (XCDR (this));
3720 if (start
>= 0 && start
<= end
&& end
< 256)
3721 while (start
<= end
)
3722 coding
->spec
.ccl
.valid_codes
[start
++] = 1;
3727 coding
->common_flags
|= CODING_REQUIRE_FLUSHING_MASK
;
3728 coding
->spec
.ccl
.cr_carryover
= 0;
3729 coding
->spec
.ccl
.eight_bit_carryover
[0] = 0;
3733 coding
->type
= coding_type_raw_text
;
3737 goto label_invalid_coding_system
;
3741 label_invalid_coding_system
:
3742 coding
->type
= coding_type_no_conversion
;
3743 coding
->category_idx
= CODING_CATEGORY_IDX_BINARY
;
3744 coding
->common_flags
= 0;
3745 coding
->eol_type
= CODING_EOL_LF
;
3746 coding
->pre_write_conversion
= coding
->post_read_conversion
= Qnil
;
3750 /* Free memory blocks allocated for storing composition information. */
3753 coding_free_composition_data (coding
)
3754 struct coding_system
*coding
;
3756 struct composition_data
*cmp_data
= coding
->cmp_data
, *next
;
3760 /* Memory blocks are chained. At first, rewind to the first, then,
3761 free blocks one by one. */
3762 while (cmp_data
->prev
)
3763 cmp_data
= cmp_data
->prev
;
3766 next
= cmp_data
->next
;
3770 coding
->cmp_data
= NULL
;
3773 /* Set `char_offset' member of all memory blocks pointed by
3774 coding->cmp_data to POS. */
3777 coding_adjust_composition_offset (coding
, pos
)
3778 struct coding_system
*coding
;
3781 struct composition_data
*cmp_data
;
3783 for (cmp_data
= coding
->cmp_data
; cmp_data
; cmp_data
= cmp_data
->next
)
3784 cmp_data
->char_offset
= pos
;
3787 /* Setup raw-text or one of its subsidiaries in the structure
3788 coding_system CODING according to the already setup value eol_type
3789 in CODING. CODING should be setup for some coding system in
3793 setup_raw_text_coding_system (coding
)
3794 struct coding_system
*coding
;
3796 if (coding
->type
!= coding_type_raw_text
)
3798 coding
->symbol
= Qraw_text
;
3799 coding
->type
= coding_type_raw_text
;
3800 if (coding
->eol_type
!= CODING_EOL_UNDECIDED
)
3802 Lisp_Object subsidiaries
;
3803 subsidiaries
= Fget (Qraw_text
, Qeol_type
);
3805 if (VECTORP (subsidiaries
)
3806 && XVECTOR (subsidiaries
)->size
== 3)
3808 = XVECTOR (subsidiaries
)->contents
[coding
->eol_type
];
3810 setup_coding_system (coding
->symbol
, coding
);
3815 /* Emacs has a mechanism to automatically detect a coding system if it
3816 is one of Emacs' internal format, ISO2022, SJIS, and BIG5. But,
3817 it's impossible to distinguish some coding systems accurately
3818 because they use the same range of codes. So, at first, coding
3819 systems are categorized into 7, those are:
3821 o coding-category-emacs-mule
3823 The category for a coding system which has the same code range
3824 as Emacs' internal format. Assigned the coding-system (Lisp
3825 symbol) `emacs-mule' by default.
3827 o coding-category-sjis
3829 The category for a coding system which has the same code range
3830 as SJIS. Assigned the coding-system (Lisp
3831 symbol) `japanese-shift-jis' by default.
3833 o coding-category-iso-7
3835 The category for a coding system which has the same code range
3836 as ISO2022 of 7-bit environment. This doesn't use any locking
3837 shift and single shift functions. This can encode/decode all
3838 charsets. Assigned the coding-system (Lisp symbol)
3839 `iso-2022-7bit' by default.
3841 o coding-category-iso-7-tight
3843 Same as coding-category-iso-7 except that this can
3844 encode/decode only the specified charsets.
3846 o coding-category-iso-8-1
3848 The category for a coding system which has the same code range
3849 as ISO2022 of 8-bit environment and graphic plane 1 used only
3850 for DIMENSION1 charset. This doesn't use any locking shift
3851 and single shift functions. Assigned the coding-system (Lisp
3852 symbol) `iso-latin-1' by default.
3854 o coding-category-iso-8-2
3856 The category for a coding system which has the same code range
3857 as ISO2022 of 8-bit environment and graphic plane 1 used only
3858 for DIMENSION2 charset. This doesn't use any locking shift
3859 and single shift functions. Assigned the coding-system (Lisp
3860 symbol) `japanese-iso-8bit' by default.
3862 o coding-category-iso-7-else
3864 The category for a coding system which has the same code range
3865 as ISO2022 of 7-bit environment but uses locking shift or
3866 single shift functions. Assigned the coding-system (Lisp
3867 symbol) `iso-2022-7bit-lock' by default.
3869 o coding-category-iso-8-else
3871 The category for a coding system which has the same code range
3872 as ISO2022 of 8-bit environment but uses locking shift or
3873 single shift functions. Assigned the coding-system (Lisp
3874 symbol) `iso-2022-8bit-ss2' by default.
3876 o coding-category-big5
3878 The category for a coding system which has the same code range
3879 as BIG5. Assigned the coding-system (Lisp symbol)
3880 `cn-big5' by default.
3882 o coding-category-utf-8
3884 The category for a coding system which has the same code range
3885 as UTF-8 (cf. RFC2279). Assigned the coding-system (Lisp
3886 symbol) `utf-8' by default.
3888 o coding-category-utf-16-be
3890 The category for a coding system in which a text has an
3891 Unicode signature (cf. Unicode Standard) in the order of BIG
3892 endian at the head. Assigned the coding-system (Lisp symbol)
3893 `utf-16-be' by default.
3895 o coding-category-utf-16-le
3897 The category for a coding system in which a text has an
3898 Unicode signature (cf. Unicode Standard) in the order of
3899 LITTLE endian at the head. Assigned the coding-system (Lisp
3900 symbol) `utf-16-le' by default.
3902 o coding-category-ccl
3904 The category for a coding system of which encoder/decoder is
3905 written in CCL programs. The default value is nil, i.e., no
3906 coding system is assigned.
3908 o coding-category-binary
3910 The category for a coding system not categorized in any of the
3911 above. Assigned the coding-system (Lisp symbol)
3912 `no-conversion' by default.
3914 Each of them is a Lisp symbol and the value is an actual
3915 `coding-system' (this is also a Lisp symbol) assigned by a user.
3916 What Emacs does actually is to detect a category of coding system.
3917 Then, it uses a `coding-system' assigned to it. If Emacs can't
3918 decide a single possible category, it selects a category of the
3919 highest priority. Priorities of categories are also specified by a
3920 user in a Lisp variable `coding-category-list'.
3925 int ascii_skip_code
[256];
3927 /* Detect how a text of length SRC_BYTES pointed by SOURCE is encoded.
3928 If it detects possible coding systems, return an integer in which
3929 appropriate flag bits are set. Flag bits are defined by macros
3930 CODING_CATEGORY_MASK_XXX in `coding.h'. If PRIORITIES is non-NULL,
3931 it should point the table `coding_priorities'. In that case, only
3932 the flag bit for a coding system of the highest priority is set in
3933 the returned value. If MULTIBYTEP is nonzero, 8-bit codes of the
3934 range 0x80..0x9F are in multibyte form.
3936 How many ASCII characters are at the head is returned as *SKIP. */
3939 detect_coding_mask (source
, src_bytes
, priorities
, skip
, multibytep
)
3940 unsigned char *source
;
3941 int src_bytes
, *priorities
, *skip
;
3944 register unsigned char c
;
3945 unsigned char *src
= source
, *src_end
= source
+ src_bytes
;
3946 unsigned int mask
, utf16_examined_p
, iso2022_examined_p
;
3949 /* At first, skip all ASCII characters and control characters except
3950 for three ISO2022 specific control characters. */
3951 ascii_skip_code
[ISO_CODE_SO
] = 0;
3952 ascii_skip_code
[ISO_CODE_SI
] = 0;
3953 ascii_skip_code
[ISO_CODE_ESC
] = 0;
3955 label_loop_detect_coding
:
3956 while (src
< src_end
&& ascii_skip_code
[*src
]) src
++;
3957 *skip
= src
- source
;
3960 /* We found nothing other than ASCII. There's nothing to do. */
3964 /* The text seems to be encoded in some multilingual coding system.
3965 Now, try to find in which coding system the text is encoded. */
3968 /* i.e. (c == ISO_CODE_ESC || c == ISO_CODE_SI || c == ISO_CODE_SO) */
3969 /* C is an ISO2022 specific control code of C0. */
3970 mask
= detect_coding_iso2022 (src
, src_end
, multibytep
);
3973 /* No valid ISO2022 code follows C. Try again. */
3975 if (c
== ISO_CODE_ESC
)
3976 ascii_skip_code
[ISO_CODE_ESC
] = 1;
3978 ascii_skip_code
[ISO_CODE_SO
] = ascii_skip_code
[ISO_CODE_SI
] = 1;
3979 goto label_loop_detect_coding
;
3983 for (i
= 0; i
< CODING_CATEGORY_IDX_MAX
; i
++)
3985 if (mask
& priorities
[i
])
3986 return priorities
[i
];
3988 return CODING_CATEGORY_MASK_RAW_TEXT
;
3995 if (multibytep
&& c
== LEADING_CODE_8_BIT_CONTROL
)
4000 /* C is the first byte of SJIS character code,
4001 or a leading-code of Emacs' internal format (emacs-mule),
4002 or the first byte of UTF-16. */
4003 try = (CODING_CATEGORY_MASK_SJIS
4004 | CODING_CATEGORY_MASK_EMACS_MULE
4005 | CODING_CATEGORY_MASK_UTF_16_BE
4006 | CODING_CATEGORY_MASK_UTF_16_LE
);
4008 /* Or, if C is a special latin extra code,
4009 or is an ISO2022 specific control code of C1 (SS2 or SS3),
4010 or is an ISO2022 control-sequence-introducer (CSI),
4011 we should also consider the possibility of ISO2022 codings. */
4012 if ((VECTORP (Vlatin_extra_code_table
)
4013 && !NILP (XVECTOR (Vlatin_extra_code_table
)->contents
[c
]))
4014 || (c
== ISO_CODE_SS2
|| c
== ISO_CODE_SS3
)
4015 || (c
== ISO_CODE_CSI
4018 || ((*src
== '0' || *src
== '1' || *src
== '2')
4019 && src
+ 1 < src_end
4020 && src
[1] == ']')))))
4021 try |= (CODING_CATEGORY_MASK_ISO_8_ELSE
4022 | CODING_CATEGORY_MASK_ISO_8BIT
);
4025 /* C is a character of ISO2022 in graphic plane right,
4026 or a SJIS's 1-byte character code (i.e. JISX0201),
4027 or the first byte of BIG5's 2-byte code,
4028 or the first byte of UTF-8/16. */
4029 try = (CODING_CATEGORY_MASK_ISO_8_ELSE
4030 | CODING_CATEGORY_MASK_ISO_8BIT
4031 | CODING_CATEGORY_MASK_SJIS
4032 | CODING_CATEGORY_MASK_BIG5
4033 | CODING_CATEGORY_MASK_UTF_8
4034 | CODING_CATEGORY_MASK_UTF_16_BE
4035 | CODING_CATEGORY_MASK_UTF_16_LE
);
4037 /* Or, we may have to consider the possibility of CCL. */
4038 if (coding_system_table
[CODING_CATEGORY_IDX_CCL
]
4039 && (coding_system_table
[CODING_CATEGORY_IDX_CCL
]
4040 ->spec
.ccl
.valid_codes
)[c
])
4041 try |= CODING_CATEGORY_MASK_CCL
;
4044 utf16_examined_p
= iso2022_examined_p
= 0;
4047 for (i
= 0; i
< CODING_CATEGORY_IDX_MAX
; i
++)
4049 if (!iso2022_examined_p
4050 && (priorities
[i
] & try & CODING_CATEGORY_MASK_ISO
))
4052 mask
|= detect_coding_iso2022 (src
, src_end
, multibytep
);
4053 iso2022_examined_p
= 1;
4055 else if (priorities
[i
] & try & CODING_CATEGORY_MASK_SJIS
)
4056 mask
|= detect_coding_sjis (src
, src_end
, multibytep
);
4057 else if (priorities
[i
] & try & CODING_CATEGORY_MASK_UTF_8
)
4058 mask
|= detect_coding_utf_8 (src
, src_end
, multibytep
);
4059 else if (!utf16_examined_p
4060 && (priorities
[i
] & try &
4061 CODING_CATEGORY_MASK_UTF_16_BE_LE
))
4063 mask
|= detect_coding_utf_16 (src
, src_end
, multibytep
);
4064 utf16_examined_p
= 1;
4066 else if (priorities
[i
] & try & CODING_CATEGORY_MASK_BIG5
)
4067 mask
|= detect_coding_big5 (src
, src_end
, multibytep
);
4068 else if (priorities
[i
] & try & CODING_CATEGORY_MASK_EMACS_MULE
)
4069 mask
|= detect_coding_emacs_mule (src
, src_end
, multibytep
);
4070 else if (priorities
[i
] & try & CODING_CATEGORY_MASK_CCL
)
4071 mask
|= detect_coding_ccl (src
, src_end
, multibytep
);
4072 else if (priorities
[i
] & CODING_CATEGORY_MASK_RAW_TEXT
)
4073 mask
|= CODING_CATEGORY_MASK_RAW_TEXT
;
4074 else if (priorities
[i
] & CODING_CATEGORY_MASK_BINARY
)
4075 mask
|= CODING_CATEGORY_MASK_BINARY
;
4076 if (mask
& priorities
[i
])
4077 return priorities
[i
];
4079 return CODING_CATEGORY_MASK_RAW_TEXT
;
4081 if (try & CODING_CATEGORY_MASK_ISO
)
4082 mask
|= detect_coding_iso2022 (src
, src_end
, multibytep
);
4083 if (try & CODING_CATEGORY_MASK_SJIS
)
4084 mask
|= detect_coding_sjis (src
, src_end
, multibytep
);
4085 if (try & CODING_CATEGORY_MASK_BIG5
)
4086 mask
|= detect_coding_big5 (src
, src_end
, multibytep
);
4087 if (try & CODING_CATEGORY_MASK_UTF_8
)
4088 mask
|= detect_coding_utf_8 (src
, src_end
, multibytep
);
4089 if (try & CODING_CATEGORY_MASK_UTF_16_BE_LE
)
4090 mask
|= detect_coding_utf_16 (src
, src_end
, multibytep
);
4091 if (try & CODING_CATEGORY_MASK_EMACS_MULE
)
4092 mask
|= detect_coding_emacs_mule (src
, src_end
, multibytep
);
4093 if (try & CODING_CATEGORY_MASK_CCL
)
4094 mask
|= detect_coding_ccl (src
, src_end
, multibytep
);
4096 return (mask
| CODING_CATEGORY_MASK_RAW_TEXT
| CODING_CATEGORY_MASK_BINARY
);
4099 /* Detect how a text of length SRC_BYTES pointed by SRC is encoded.
4100 The information of the detected coding system is set in CODING. */
4103 detect_coding (coding
, src
, src_bytes
)
4104 struct coding_system
*coding
;
4105 const unsigned char *src
;
4112 val
= Vcoding_category_list
;
4113 mask
= detect_coding_mask (src
, src_bytes
, coding_priorities
, &skip
,
4114 coding
->src_multibyte
);
4115 coding
->heading_ascii
= skip
;
4119 /* We found a single coding system of the highest priority in MASK. */
4121 while (mask
&& ! (mask
& 1)) mask
>>= 1, idx
++;
4123 idx
= CODING_CATEGORY_IDX_RAW_TEXT
;
4125 val
= SYMBOL_VALUE (XVECTOR (Vcoding_category_table
)->contents
[idx
]);
4127 if (coding
->eol_type
!= CODING_EOL_UNDECIDED
)
4131 tmp
= Fget (val
, Qeol_type
);
4133 val
= XVECTOR (tmp
)->contents
[coding
->eol_type
];
4136 /* Setup this new coding system while preserving some slots. */
4138 int src_multibyte
= coding
->src_multibyte
;
4139 int dst_multibyte
= coding
->dst_multibyte
;
4141 setup_coding_system (val
, coding
);
4142 coding
->src_multibyte
= src_multibyte
;
4143 coding
->dst_multibyte
= dst_multibyte
;
4144 coding
->heading_ascii
= skip
;
4148 /* Detect how end-of-line of a text of length SRC_BYTES pointed by
4149 SOURCE is encoded. Return one of CODING_EOL_LF, CODING_EOL_CRLF,
4150 CODING_EOL_CR, and CODING_EOL_UNDECIDED.
4152 How many non-eol characters are at the head is returned as *SKIP. */
4154 #define MAX_EOL_CHECK_COUNT 3
4157 detect_eol_type (source
, src_bytes
, skip
)
4158 unsigned char *source
;
4159 int src_bytes
, *skip
;
4161 unsigned char *src
= source
, *src_end
= src
+ src_bytes
;
4163 int total
= 0; /* How many end-of-lines are found so far. */
4164 int eol_type
= CODING_EOL_UNDECIDED
;
4169 while (src
< src_end
&& total
< MAX_EOL_CHECK_COUNT
)
4172 if (c
== '\n' || c
== '\r')
4175 *skip
= src
- 1 - source
;
4178 this_eol_type
= CODING_EOL_LF
;
4179 else if (src
>= src_end
|| *src
!= '\n')
4180 this_eol_type
= CODING_EOL_CR
;
4182 this_eol_type
= CODING_EOL_CRLF
, src
++;
4184 if (eol_type
== CODING_EOL_UNDECIDED
)
4185 /* This is the first end-of-line. */
4186 eol_type
= this_eol_type
;
4187 else if (eol_type
!= this_eol_type
)
4189 /* The found type is different from what found before. */
4190 eol_type
= CODING_EOL_INCONSISTENT
;
4197 *skip
= src_end
- source
;
4201 /* Like detect_eol_type, but detect EOL type in 2-octet
4202 big-endian/little-endian format for coding systems utf-16-be and
4206 detect_eol_type_in_2_octet_form (source
, src_bytes
, skip
, big_endian_p
)
4207 unsigned char *source
;
4208 int src_bytes
, *skip
, big_endian_p
;
4210 unsigned char *src
= source
, *src_end
= src
+ src_bytes
;
4211 unsigned int c1
, c2
;
4212 int total
= 0; /* How many end-of-lines are found so far. */
4213 int eol_type
= CODING_EOL_UNDECIDED
;
4224 while ((src
+ 1) < src_end
&& total
< MAX_EOL_CHECK_COUNT
)
4226 c1
= (src
[msb
] << 8) | (src
[lsb
]);
4229 if (c1
== '\n' || c1
== '\r')
4232 *skip
= src
- 2 - source
;
4236 this_eol_type
= CODING_EOL_LF
;
4240 if ((src
+ 1) >= src_end
)
4242 this_eol_type
= CODING_EOL_CR
;
4246 c2
= (src
[msb
] << 8) | (src
[lsb
]);
4248 this_eol_type
= CODING_EOL_CRLF
, src
+= 2;
4250 this_eol_type
= CODING_EOL_CR
;
4254 if (eol_type
== CODING_EOL_UNDECIDED
)
4255 /* This is the first end-of-line. */
4256 eol_type
= this_eol_type
;
4257 else if (eol_type
!= this_eol_type
)
4259 /* The found type is different from what found before. */
4260 eol_type
= CODING_EOL_INCONSISTENT
;
4267 *skip
= src_end
- source
;
4271 /* Detect how end-of-line of a text of length SRC_BYTES pointed by SRC
4272 is encoded. If it detects an appropriate format of end-of-line, it
4273 sets the information in *CODING. */
4276 detect_eol (coding
, src
, src_bytes
)
4277 struct coding_system
*coding
;
4278 const unsigned char *src
;
4285 switch (coding
->category_idx
)
4287 case CODING_CATEGORY_IDX_UTF_16_BE
:
4288 eol_type
= detect_eol_type_in_2_octet_form (src
, src_bytes
, &skip
, 1);
4290 case CODING_CATEGORY_IDX_UTF_16_LE
:
4291 eol_type
= detect_eol_type_in_2_octet_form (src
, src_bytes
, &skip
, 0);
4294 eol_type
= detect_eol_type (src
, src_bytes
, &skip
);
4298 if (coding
->heading_ascii
> skip
)
4299 coding
->heading_ascii
= skip
;
4301 skip
= coding
->heading_ascii
;
4303 if (eol_type
== CODING_EOL_UNDECIDED
)
4305 if (eol_type
== CODING_EOL_INCONSISTENT
)
4308 /* This code is suppressed until we find a better way to
4309 distinguish raw text file and binary file. */
4311 /* If we have already detected that the coding is raw-text, the
4312 coding should actually be no-conversion. */
4313 if (coding
->type
== coding_type_raw_text
)
4315 setup_coding_system (Qno_conversion
, coding
);
4318 /* Else, let's decode only text code anyway. */
4320 eol_type
= CODING_EOL_LF
;
4323 val
= Fget (coding
->symbol
, Qeol_type
);
4324 if (VECTORP (val
) && XVECTOR (val
)->size
== 3)
4326 int src_multibyte
= coding
->src_multibyte
;
4327 int dst_multibyte
= coding
->dst_multibyte
;
4328 struct composition_data
*cmp_data
= coding
->cmp_data
;
4330 setup_coding_system (XVECTOR (val
)->contents
[eol_type
], coding
);
4331 coding
->src_multibyte
= src_multibyte
;
4332 coding
->dst_multibyte
= dst_multibyte
;
4333 coding
->heading_ascii
= skip
;
4334 coding
->cmp_data
= cmp_data
;
4338 #define CONVERSION_BUFFER_EXTRA_ROOM 256
4340 #define DECODING_BUFFER_MAG(coding) \
4341 (coding->type == coding_type_iso2022 \
4343 : (coding->type == coding_type_ccl \
4344 ? coding->spec.ccl.decoder.buf_magnification \
4347 /* Return maximum size (bytes) of a buffer enough for decoding
4348 SRC_BYTES of text encoded in CODING. */
4351 decoding_buffer_size (coding
, src_bytes
)
4352 struct coding_system
*coding
;
4355 return (src_bytes
* DECODING_BUFFER_MAG (coding
)
4356 + CONVERSION_BUFFER_EXTRA_ROOM
);
4359 /* Return maximum size (bytes) of a buffer enough for encoding
4360 SRC_BYTES of text to CODING. */
4363 encoding_buffer_size (coding
, src_bytes
)
4364 struct coding_system
*coding
;
4369 if (coding
->type
== coding_type_ccl
)
4370 magnification
= coding
->spec
.ccl
.encoder
.buf_magnification
;
4371 else if (CODING_REQUIRE_ENCODING (coding
))
4376 return (src_bytes
* magnification
+ CONVERSION_BUFFER_EXTRA_ROOM
);
4379 /* Working buffer for code conversion. */
4380 struct conversion_buffer
4382 int size
; /* size of data. */
4383 int on_stack
; /* 1 if allocated by alloca. */
4384 unsigned char *data
;
4387 /* Don't use alloca for allocating memory space larger than this, lest
4388 we overflow their stack. */
4389 #define MAX_ALLOCA 16*1024
4391 /* Allocate LEN bytes of memory for BUF (struct conversion_buffer). */
4392 #define allocate_conversion_buffer(buf, len) \
4394 if (len < MAX_ALLOCA) \
4396 buf.data = (unsigned char *) alloca (len); \
4401 buf.data = (unsigned char *) xmalloc (len); \
4407 /* Double the allocated memory for *BUF. */
4409 extend_conversion_buffer (buf
)
4410 struct conversion_buffer
*buf
;
4414 unsigned char *save
= buf
->data
;
4415 buf
->data
= (unsigned char *) xmalloc (buf
->size
* 2);
4416 bcopy (save
, buf
->data
, buf
->size
);
4421 buf
->data
= (unsigned char *) xrealloc (buf
->data
, buf
->size
* 2);
4426 /* Free the allocated memory for BUF if it is not on stack. */
4428 free_conversion_buffer (buf
)
4429 struct conversion_buffer
*buf
;
4436 ccl_coding_driver (coding
, source
, destination
, src_bytes
, dst_bytes
, encodep
)
4437 struct coding_system
*coding
;
4438 unsigned char *source
, *destination
;
4439 int src_bytes
, dst_bytes
, encodep
;
4441 struct ccl_program
*ccl
4442 = encodep
? &coding
->spec
.ccl
.encoder
: &coding
->spec
.ccl
.decoder
;
4443 unsigned char *dst
= destination
;
4445 ccl
->suppress_error
= coding
->suppress_error
;
4446 ccl
->last_block
= coding
->mode
& CODING_MODE_LAST_BLOCK
;
4449 /* On encoding, EOL format is converted within ccl_driver. For
4450 that, setup proper information in the structure CCL. */
4451 ccl
->eol_type
= coding
->eol_type
;
4452 if (ccl
->eol_type
==CODING_EOL_UNDECIDED
)
4453 ccl
->eol_type
= CODING_EOL_LF
;
4454 ccl
->cr_consumed
= coding
->spec
.ccl
.cr_carryover
;
4456 ccl
->multibyte
= coding
->src_multibyte
;
4457 if (coding
->spec
.ccl
.eight_bit_carryover
[0] != 0)
4459 /* Move carryover bytes to DESTINATION. */
4460 unsigned char *p
= coding
->spec
.ccl
.eight_bit_carryover
;
4463 coding
->spec
.ccl
.eight_bit_carryover
[0] = 0;
4465 dst_bytes
-= dst
- destination
;
4468 coding
->produced
= (ccl_driver (ccl
, source
, dst
, src_bytes
, dst_bytes
,
4469 &(coding
->consumed
))
4470 + dst
- destination
);
4474 coding
->produced_char
= coding
->produced
;
4475 coding
->spec
.ccl
.cr_carryover
= ccl
->cr_consumed
;
4477 else if (!ccl
->eight_bit_control
)
4479 /* The produced bytes forms a valid multibyte sequence. */
4480 coding
->produced_char
4481 = multibyte_chars_in_text (destination
, coding
->produced
);
4482 coding
->spec
.ccl
.eight_bit_carryover
[0] = 0;
4486 /* On decoding, the destination should always multibyte. But,
4487 CCL program might have been generated an invalid multibyte
4488 sequence. Here we make such a sequence valid as
4491 = dst_bytes
? dst_bytes
: source
+ coding
->consumed
- destination
;
4493 if ((coding
->consumed
< src_bytes
4494 || !ccl
->last_block
)
4495 && coding
->produced
>= 1
4496 && destination
[coding
->produced
- 1] >= 0x80)
4498 /* We should not convert the tailing 8-bit codes to
4499 multibyte form even if they doesn't form a valid
4500 multibyte sequence. They may form a valid sequence in
4504 if (destination
[coding
->produced
- 1] < 0xA0)
4506 else if (coding
->produced
>= 2)
4508 if (destination
[coding
->produced
- 2] >= 0x80)
4510 if (destination
[coding
->produced
- 2] < 0xA0)
4512 else if (coding
->produced
>= 3
4513 && destination
[coding
->produced
- 3] >= 0x80
4514 && destination
[coding
->produced
- 3] < 0xA0)
4520 BCOPY_SHORT (destination
+ coding
->produced
- carryover
,
4521 coding
->spec
.ccl
.eight_bit_carryover
,
4523 coding
->spec
.ccl
.eight_bit_carryover
[carryover
] = 0;
4524 coding
->produced
-= carryover
;
4527 coding
->produced
= str_as_multibyte (destination
, bytes
,
4529 &(coding
->produced_char
));
4532 switch (ccl
->status
)
4534 case CCL_STAT_SUSPEND_BY_SRC
:
4535 coding
->result
= CODING_FINISH_INSUFFICIENT_SRC
;
4537 case CCL_STAT_SUSPEND_BY_DST
:
4538 coding
->result
= CODING_FINISH_INSUFFICIENT_DST
;
4541 case CCL_STAT_INVALID_CMD
:
4542 coding
->result
= CODING_FINISH_INTERRUPT
;
4545 coding
->result
= CODING_FINISH_NORMAL
;
4548 return coding
->result
;
4551 /* Decode EOL format of the text at PTR of BYTES length destructively
4552 according to CODING->eol_type. This is called after the CCL
4553 program produced a decoded text at PTR. If we do CRLF->LF
4554 conversion, update CODING->produced and CODING->produced_char. */
4557 decode_eol_post_ccl (coding
, ptr
, bytes
)
4558 struct coding_system
*coding
;
4562 Lisp_Object val
, saved_coding_symbol
;
4563 unsigned char *pend
= ptr
+ bytes
;
4566 /* Remember the current coding system symbol. We set it back when
4567 an inconsistent EOL is found so that `last-coding-system-used' is
4568 set to the coding system that doesn't specify EOL conversion. */
4569 saved_coding_symbol
= coding
->symbol
;
4571 coding
->spec
.ccl
.cr_carryover
= 0;
4572 if (coding
->eol_type
== CODING_EOL_UNDECIDED
)
4574 /* Here, to avoid the call of setup_coding_system, we directly
4575 call detect_eol_type. */
4576 coding
->eol_type
= detect_eol_type (ptr
, bytes
, &dummy
);
4577 if (coding
->eol_type
== CODING_EOL_INCONSISTENT
)
4578 coding
->eol_type
= CODING_EOL_LF
;
4579 if (coding
->eol_type
!= CODING_EOL_UNDECIDED
)
4581 val
= Fget (coding
->symbol
, Qeol_type
);
4582 if (VECTORP (val
) && XVECTOR (val
)->size
== 3)
4583 coding
->symbol
= XVECTOR (val
)->contents
[coding
->eol_type
];
4585 coding
->mode
|= CODING_MODE_INHIBIT_INCONSISTENT_EOL
;
4588 if (coding
->eol_type
== CODING_EOL_LF
4589 || coding
->eol_type
== CODING_EOL_UNDECIDED
)
4591 /* We have nothing to do. */
4594 else if (coding
->eol_type
== CODING_EOL_CRLF
)
4596 unsigned char *pstart
= ptr
, *p
= ptr
;
4598 if (! (coding
->mode
& CODING_MODE_LAST_BLOCK
)
4599 && *(pend
- 1) == '\r')
4601 /* If the last character is CR, we can't handle it here
4602 because LF will be in the not-yet-decoded source text.
4603 Record that the CR is not yet processed. */
4604 coding
->spec
.ccl
.cr_carryover
= 1;
4606 coding
->produced_char
--;
4613 if (ptr
+ 1 < pend
&& *(ptr
+ 1) == '\n')
4620 if (coding
->mode
& CODING_MODE_INHIBIT_INCONSISTENT_EOL
)
4621 goto undo_eol_conversion
;
4625 else if (*ptr
== '\n'
4626 && coding
->mode
& CODING_MODE_INHIBIT_INCONSISTENT_EOL
)
4627 goto undo_eol_conversion
;
4632 undo_eol_conversion
:
4633 /* We have faced with inconsistent EOL format at PTR.
4634 Convert all LFs before PTR back to CRLFs. */
4635 for (p
--, ptr
--; p
>= pstart
; p
--)
4638 *ptr
-- = '\n', *ptr
-- = '\r';
4642 /* If carryover is recorded, cancel it because we don't
4643 convert CRLF anymore. */
4644 if (coding
->spec
.ccl
.cr_carryover
)
4646 coding
->spec
.ccl
.cr_carryover
= 0;
4648 coding
->produced_char
++;
4652 coding
->eol_type
= CODING_EOL_LF
;
4653 coding
->symbol
= saved_coding_symbol
;
4657 /* As each two-byte sequence CRLF was converted to LF, (PEND
4658 - P) is the number of deleted characters. */
4659 coding
->produced
-= pend
- p
;
4660 coding
->produced_char
-= pend
- p
;
4663 else /* i.e. coding->eol_type == CODING_EOL_CR */
4665 unsigned char *p
= ptr
;
4667 for (; ptr
< pend
; ptr
++)
4671 else if (*ptr
== '\n'
4672 && coding
->mode
& CODING_MODE_INHIBIT_INCONSISTENT_EOL
)
4674 for (; p
< ptr
; p
++)
4680 coding
->eol_type
= CODING_EOL_LF
;
4681 coding
->symbol
= saved_coding_symbol
;
4687 /* See "GENERAL NOTES about `decode_coding_XXX ()' functions". Before
4688 decoding, it may detect coding system and format of end-of-line if
4689 those are not yet decided. The source should be unibyte, the
4690 result is multibyte if CODING->dst_multibyte is nonzero, else
4694 decode_coding (coding
, source
, destination
, src_bytes
, dst_bytes
)
4695 struct coding_system
*coding
;
4696 const unsigned char *source
;
4697 unsigned char *destination
;
4698 int src_bytes
, dst_bytes
;
4702 if (coding
->type
== coding_type_undecided
)
4703 detect_coding (coding
, source
, src_bytes
);
4705 if (coding
->eol_type
== CODING_EOL_UNDECIDED
4706 && coding
->type
!= coding_type_ccl
)
4708 detect_eol (coding
, source
, src_bytes
);
4709 /* We had better recover the original eol format if we
4710 encounter an inconsistent eol format while decoding. */
4711 coding
->mode
|= CODING_MODE_INHIBIT_INCONSISTENT_EOL
;
4714 coding
->produced
= coding
->produced_char
= 0;
4715 coding
->consumed
= coding
->consumed_char
= 0;
4717 coding
->result
= CODING_FINISH_NORMAL
;
4719 switch (coding
->type
)
4721 case coding_type_sjis
:
4722 decode_coding_sjis_big5 (coding
, source
, destination
,
4723 src_bytes
, dst_bytes
, 1);
4726 case coding_type_iso2022
:
4727 decode_coding_iso2022 (coding
, source
, destination
,
4728 src_bytes
, dst_bytes
);
4731 case coding_type_big5
:
4732 decode_coding_sjis_big5 (coding
, source
, destination
,
4733 src_bytes
, dst_bytes
, 0);
4736 case coding_type_emacs_mule
:
4737 decode_coding_emacs_mule (coding
, source
, destination
,
4738 src_bytes
, dst_bytes
);
4741 case coding_type_ccl
:
4742 if (coding
->spec
.ccl
.cr_carryover
)
4744 /* Put the CR which was not processed by the previous call
4745 of decode_eol_post_ccl in DESTINATION. It will be
4746 decoded together with the following LF by the call to
4747 decode_eol_post_ccl below. */
4748 *destination
= '\r';
4750 coding
->produced_char
++;
4752 extra
= coding
->spec
.ccl
.cr_carryover
;
4754 ccl_coding_driver (coding
, source
, destination
+ extra
,
4755 src_bytes
, dst_bytes
, 0);
4756 if (coding
->eol_type
!= CODING_EOL_LF
)
4758 coding
->produced
+= extra
;
4759 coding
->produced_char
+= extra
;
4760 decode_eol_post_ccl (coding
, destination
, coding
->produced
);
4765 decode_eol (coding
, source
, destination
, src_bytes
, dst_bytes
);
4768 if (coding
->result
== CODING_FINISH_INSUFFICIENT_SRC
4769 && coding
->mode
& CODING_MODE_LAST_BLOCK
4770 && coding
->consumed
== src_bytes
)
4771 coding
->result
= CODING_FINISH_NORMAL
;
4773 if (coding
->mode
& CODING_MODE_LAST_BLOCK
4774 && coding
->result
== CODING_FINISH_INSUFFICIENT_SRC
)
4776 const unsigned char *src
= source
+ coding
->consumed
;
4777 unsigned char *dst
= destination
+ coding
->produced
;
4779 src_bytes
-= coding
->consumed
;
4781 if (COMPOSING_P (coding
))
4782 DECODE_COMPOSITION_END ('1');
4786 dst
+= CHAR_STRING (c
, dst
);
4787 coding
->produced_char
++;
4789 coding
->consumed
= coding
->consumed_char
= src
- source
;
4790 coding
->produced
= dst
- destination
;
4791 coding
->result
= CODING_FINISH_NORMAL
;
4794 if (!coding
->dst_multibyte
)
4796 coding
->produced
= str_as_unibyte (destination
, coding
->produced
);
4797 coding
->produced_char
= coding
->produced
;
4800 return coding
->result
;
4803 /* See "GENERAL NOTES about `encode_coding_XXX ()' functions". The
4804 multibyteness of the source is CODING->src_multibyte, the
4805 multibyteness of the result is always unibyte. */
4808 encode_coding (coding
, source
, destination
, src_bytes
, dst_bytes
)
4809 struct coding_system
*coding
;
4810 const unsigned char *source
;
4811 unsigned char *destination
;
4812 int src_bytes
, dst_bytes
;
4814 coding
->produced
= coding
->produced_char
= 0;
4815 coding
->consumed
= coding
->consumed_char
= 0;
4817 coding
->result
= CODING_FINISH_NORMAL
;
4819 switch (coding
->type
)
4821 case coding_type_sjis
:
4822 encode_coding_sjis_big5 (coding
, source
, destination
,
4823 src_bytes
, dst_bytes
, 1);
4826 case coding_type_iso2022
:
4827 encode_coding_iso2022 (coding
, source
, destination
,
4828 src_bytes
, dst_bytes
);
4831 case coding_type_big5
:
4832 encode_coding_sjis_big5 (coding
, source
, destination
,
4833 src_bytes
, dst_bytes
, 0);
4836 case coding_type_emacs_mule
:
4837 encode_coding_emacs_mule (coding
, source
, destination
,
4838 src_bytes
, dst_bytes
);
4841 case coding_type_ccl
:
4842 ccl_coding_driver (coding
, source
, destination
,
4843 src_bytes
, dst_bytes
, 1);
4847 encode_eol (coding
, source
, destination
, src_bytes
, dst_bytes
);
4850 if (coding
->mode
& CODING_MODE_LAST_BLOCK
4851 && coding
->result
== CODING_FINISH_INSUFFICIENT_SRC
)
4853 const unsigned char *src
= source
+ coding
->consumed
;
4854 unsigned char *dst
= destination
+ coding
->produced
;
4856 if (coding
->type
== coding_type_iso2022
)
4857 ENCODE_RESET_PLANE_AND_REGISTER
;
4858 if (COMPOSING_P (coding
))
4859 *dst
++ = ISO_CODE_ESC
, *dst
++ = '1';
4860 if (coding
->consumed
< src_bytes
)
4862 int len
= src_bytes
- coding
->consumed
;
4864 BCOPY_SHORT (src
, dst
, len
);
4865 if (coding
->src_multibyte
)
4866 len
= str_as_unibyte (dst
, len
);
4868 coding
->consumed
= src_bytes
;
4870 coding
->produced
= coding
->produced_char
= dst
- destination
;
4871 coding
->result
= CODING_FINISH_NORMAL
;
4874 if (coding
->result
== CODING_FINISH_INSUFFICIENT_SRC
4875 && coding
->consumed
== src_bytes
)
4876 coding
->result
= CODING_FINISH_NORMAL
;
4878 return coding
->result
;
4881 /* Scan text in the region between *BEG and *END (byte positions),
4882 skip characters which we don't have to decode by coding system
4883 CODING at the head and tail, then set *BEG and *END to the region
4884 of the text we actually have to convert. The caller should move
4885 the gap out of the region in advance if the region is from a
4888 If STR is not NULL, *BEG and *END are indices into STR. */
4891 shrink_decoding_region (beg
, end
, coding
, str
)
4893 struct coding_system
*coding
;
4896 unsigned char *begp_orig
, *begp
, *endp_orig
, *endp
, c
;
4898 Lisp_Object translation_table
;
4900 if (coding
->type
== coding_type_ccl
4901 || coding
->type
== coding_type_undecided
4902 || coding
->eol_type
!= CODING_EOL_LF
4903 || !NILP (coding
->post_read_conversion
)
4904 || coding
->composing
!= COMPOSITION_DISABLED
)
4906 /* We can't skip any data. */
4909 if (coding
->type
== coding_type_no_conversion
4910 || coding
->type
== coding_type_raw_text
4911 || coding
->type
== coding_type_emacs_mule
)
4913 /* We need no conversion, but don't have to skip any data here.
4914 Decoding routine handles them effectively anyway. */
4918 translation_table
= coding
->translation_table_for_decode
;
4919 if (NILP (translation_table
) && !NILP (Venable_character_translation
))
4920 translation_table
= Vstandard_translation_table_for_decode
;
4921 if (CHAR_TABLE_P (translation_table
))
4924 for (i
= 0; i
< 128; i
++)
4925 if (!NILP (CHAR_TABLE_REF (translation_table
, i
)))
4928 /* Some ASCII character should be translated. We give up
4933 if (coding
->heading_ascii
>= 0)
4934 /* Detection routine has already found how much we can skip at the
4936 *beg
+= coding
->heading_ascii
;
4940 begp_orig
= begp
= str
+ *beg
;
4941 endp_orig
= endp
= str
+ *end
;
4945 begp_orig
= begp
= BYTE_POS_ADDR (*beg
);
4946 endp_orig
= endp
= begp
+ *end
- *beg
;
4949 eol_conversion
= (coding
->eol_type
== CODING_EOL_CR
4950 || coding
->eol_type
== CODING_EOL_CRLF
);
4952 switch (coding
->type
)
4954 case coding_type_sjis
:
4955 case coding_type_big5
:
4956 /* We can skip all ASCII characters at the head. */
4957 if (coding
->heading_ascii
< 0)
4960 while (begp
< endp
&& *begp
< 0x80 && *begp
!= '\r') begp
++;
4962 while (begp
< endp
&& *begp
< 0x80) begp
++;
4964 /* We can skip all ASCII characters at the tail except for the
4965 second byte of SJIS or BIG5 code. */
4967 while (begp
< endp
&& endp
[-1] < 0x80 && endp
[-1] != '\r') endp
--;
4969 while (begp
< endp
&& endp
[-1] < 0x80) endp
--;
4970 /* Do not consider LF as ascii if preceded by CR, since that
4971 confuses eol decoding. */
4972 if (begp
< endp
&& endp
< endp_orig
&& endp
[-1] == '\r' && endp
[0] == '\n')
4974 if (begp
< endp
&& endp
< endp_orig
&& endp
[-1] >= 0x80)
4978 case coding_type_iso2022
:
4979 if (CODING_SPEC_ISO_INITIAL_DESIGNATION (coding
, 0) != CHARSET_ASCII
)
4980 /* We can't skip any data. */
4982 if (coding
->heading_ascii
< 0)
4984 /* We can skip all ASCII characters at the head except for a
4985 few control codes. */
4986 while (begp
< endp
&& (c
= *begp
) < 0x80
4987 && c
!= ISO_CODE_CR
&& c
!= ISO_CODE_SO
4988 && c
!= ISO_CODE_SI
&& c
!= ISO_CODE_ESC
4989 && (!eol_conversion
|| c
!= ISO_CODE_LF
))
4992 switch (coding
->category_idx
)
4994 case CODING_CATEGORY_IDX_ISO_8_1
:
4995 case CODING_CATEGORY_IDX_ISO_8_2
:
4996 /* We can skip all ASCII characters at the tail. */
4998 while (begp
< endp
&& (c
= endp
[-1]) < 0x80 && c
!= '\r') endp
--;
5000 while (begp
< endp
&& endp
[-1] < 0x80) endp
--;
5001 /* Do not consider LF as ascii if preceded by CR, since that
5002 confuses eol decoding. */
5003 if (begp
< endp
&& endp
< endp_orig
&& endp
[-1] == '\r' && endp
[0] == '\n')
5007 case CODING_CATEGORY_IDX_ISO_7
:
5008 case CODING_CATEGORY_IDX_ISO_7_TIGHT
:
5010 /* We can skip all characters at the tail except for 8-bit
5011 codes and ESC and the following 2-byte at the tail. */
5012 unsigned char *eight_bit
= NULL
;
5016 && (c
= endp
[-1]) != ISO_CODE_ESC
&& c
!= '\r')
5018 if (!eight_bit
&& c
& 0x80) eight_bit
= endp
;
5023 && (c
= endp
[-1]) != ISO_CODE_ESC
)
5025 if (!eight_bit
&& c
& 0x80) eight_bit
= endp
;
5028 /* Do not consider LF as ascii if preceded by CR, since that
5029 confuses eol decoding. */
5030 if (begp
< endp
&& endp
< endp_orig
5031 && endp
[-1] == '\r' && endp
[0] == '\n')
5033 if (begp
< endp
&& endp
[-1] == ISO_CODE_ESC
)
5035 if (endp
+ 1 < endp_orig
&& end
[0] == '(' && end
[1] == 'B')
5036 /* This is an ASCII designation sequence. We can
5037 surely skip the tail. But, if we have
5038 encountered an 8-bit code, skip only the codes
5040 endp
= eight_bit
? eight_bit
: endp
+ 2;
5042 /* Hmmm, we can't skip the tail. */
5054 *beg
+= begp
- begp_orig
;
5055 *end
+= endp
- endp_orig
;
5059 /* Like shrink_decoding_region but for encoding. */
5062 shrink_encoding_region (beg
, end
, coding
, str
)
5064 struct coding_system
*coding
;
5067 unsigned char *begp_orig
, *begp
, *endp_orig
, *endp
;
5069 Lisp_Object translation_table
;
5071 if (coding
->type
== coding_type_ccl
5072 || coding
->eol_type
== CODING_EOL_CRLF
5073 || coding
->eol_type
== CODING_EOL_CR
5074 || (coding
->cmp_data
&& coding
->cmp_data
->used
> 0))
5076 /* We can't skip any data. */
5079 if (coding
->type
== coding_type_no_conversion
5080 || coding
->type
== coding_type_raw_text
5081 || coding
->type
== coding_type_emacs_mule
5082 || coding
->type
== coding_type_undecided
)
5084 /* We need no conversion, but don't have to skip any data here.
5085 Encoding routine handles them effectively anyway. */
5089 translation_table
= coding
->translation_table_for_encode
;
5090 if (NILP (translation_table
) && !NILP (Venable_character_translation
))
5091 translation_table
= Vstandard_translation_table_for_encode
;
5092 if (CHAR_TABLE_P (translation_table
))
5095 for (i
= 0; i
< 128; i
++)
5096 if (!NILP (CHAR_TABLE_REF (translation_table
, i
)))
5099 /* Some ASCII character should be translated. We give up
5106 begp_orig
= begp
= str
+ *beg
;
5107 endp_orig
= endp
= str
+ *end
;
5111 begp_orig
= begp
= BYTE_POS_ADDR (*beg
);
5112 endp_orig
= endp
= begp
+ *end
- *beg
;
5115 eol_conversion
= (coding
->eol_type
== CODING_EOL_CR
5116 || coding
->eol_type
== CODING_EOL_CRLF
);
5118 /* Here, we don't have to check coding->pre_write_conversion because
5119 the caller is expected to have handled it already. */
5120 switch (coding
->type
)
5122 case coding_type_iso2022
:
5123 if (CODING_SPEC_ISO_INITIAL_DESIGNATION (coding
, 0) != CHARSET_ASCII
)
5124 /* We can't skip any data. */
5126 if (coding
->flags
& CODING_FLAG_ISO_DESIGNATE_AT_BOL
)
5128 unsigned char *bol
= begp
;
5129 while (begp
< endp
&& *begp
< 0x80)
5132 if (begp
[-1] == '\n')
5136 goto label_skip_tail
;
5140 case coding_type_sjis
:
5141 case coding_type_big5
:
5142 /* We can skip all ASCII characters at the head and tail. */
5144 while (begp
< endp
&& *begp
< 0x80 && *begp
!= '\n') begp
++;
5146 while (begp
< endp
&& *begp
< 0x80) begp
++;
5149 while (begp
< endp
&& endp
[-1] < 0x80 && endp
[-1] != '\n') endp
--;
5151 while (begp
< endp
&& *(endp
- 1) < 0x80) endp
--;
5158 *beg
+= begp
- begp_orig
;
5159 *end
+= endp
- endp_orig
;
5163 /* As shrinking conversion region requires some overhead, we don't try
5164 shrinking if the length of conversion region is less than this
5166 static int shrink_conversion_region_threshhold
= 1024;
5168 #define SHRINK_CONVERSION_REGION(beg, end, coding, str, encodep) \
5170 if (*(end) - *(beg) > shrink_conversion_region_threshhold) \
5172 if (encodep) shrink_encoding_region (beg, end, coding, str); \
5173 else shrink_decoding_region (beg, end, coding, str); \
5178 code_convert_region_unwind (dummy
)
5181 inhibit_pre_post_conversion
= 0;
5185 /* Store information about all compositions in the range FROM and TO
5186 of OBJ in memory blocks pointed by CODING->cmp_data. OBJ is a
5187 buffer or a string, defaults to the current buffer. */
5190 coding_save_composition (coding
, from
, to
, obj
)
5191 struct coding_system
*coding
;
5198 if (coding
->composing
== COMPOSITION_DISABLED
)
5200 if (!coding
->cmp_data
)
5201 coding_allocate_composition_data (coding
, from
);
5202 if (!find_composition (from
, to
, &start
, &end
, &prop
, obj
)
5206 && (!find_composition (end
, to
, &start
, &end
, &prop
, obj
)
5209 coding
->composing
= COMPOSITION_NO
;
5212 if (COMPOSITION_VALID_P (start
, end
, prop
))
5214 enum composition_method method
= COMPOSITION_METHOD (prop
);
5215 if (coding
->cmp_data
->used
+ COMPOSITION_DATA_MAX_BUNCH_LENGTH
5216 >= COMPOSITION_DATA_SIZE
)
5217 coding_allocate_composition_data (coding
, from
);
5218 /* For relative composition, we remember start and end
5219 positions, for the other compositions, we also remember
5221 CODING_ADD_COMPOSITION_START (coding
, start
- from
, method
);
5222 if (method
!= COMPOSITION_RELATIVE
)
5224 /* We must store a*/
5225 Lisp_Object val
, ch
;
5227 val
= COMPOSITION_COMPONENTS (prop
);
5231 ch
= XCAR (val
), val
= XCDR (val
);
5232 CODING_ADD_COMPOSITION_COMPONENT (coding
, XINT (ch
));
5234 else if (VECTORP (val
) || STRINGP (val
))
5236 int len
= (VECTORP (val
)
5237 ? XVECTOR (val
)->size
: SCHARS (val
));
5239 for (i
= 0; i
< len
; i
++)
5242 ? Faref (val
, make_number (i
))
5243 : XVECTOR (val
)->contents
[i
]);
5244 CODING_ADD_COMPOSITION_COMPONENT (coding
, XINT (ch
));
5247 else /* INTEGERP (val) */
5248 CODING_ADD_COMPOSITION_COMPONENT (coding
, XINT (val
));
5250 CODING_ADD_COMPOSITION_END (coding
, end
- from
);
5255 && find_composition (start
, to
, &start
, &end
, &prop
, obj
)
5258 /* Make coding->cmp_data point to the first memory block. */
5259 while (coding
->cmp_data
->prev
)
5260 coding
->cmp_data
= coding
->cmp_data
->prev
;
5261 coding
->cmp_data_start
= 0;
5264 /* Reflect the saved information about compositions to OBJ.
5265 CODING->cmp_data points to a memory block for the information. OBJ
5266 is a buffer or a string, defaults to the current buffer. */
5269 coding_restore_composition (coding
, obj
)
5270 struct coding_system
*coding
;
5273 struct composition_data
*cmp_data
= coding
->cmp_data
;
5278 while (cmp_data
->prev
)
5279 cmp_data
= cmp_data
->prev
;
5285 for (i
= 0; i
< cmp_data
->used
&& cmp_data
->data
[i
] > 0;
5286 i
+= cmp_data
->data
[i
])
5288 int *data
= cmp_data
->data
+ i
;
5289 enum composition_method method
= (enum composition_method
) data
[3];
5290 Lisp_Object components
;
5292 if (method
== COMPOSITION_RELATIVE
)
5296 int len
= data
[0] - 4, j
;
5297 Lisp_Object args
[MAX_COMPOSITION_COMPONENTS
* 2 - 1];
5299 for (j
= 0; j
< len
; j
++)
5300 args
[j
] = make_number (data
[4 + j
]);
5301 components
= (method
== COMPOSITION_WITH_ALTCHARS
5302 ? Fstring (len
, args
) : Fvector (len
, args
));
5304 compose_text (data
[1], data
[2], components
, Qnil
, obj
);
5306 cmp_data
= cmp_data
->next
;
5310 /* Decode (if ENCODEP is zero) or encode (if ENCODEP is nonzero) the
5311 text from FROM to TO (byte positions are FROM_BYTE and TO_BYTE) by
5312 coding system CODING, and return the status code of code conversion
5313 (currently, this value has no meaning).
5315 How many characters (and bytes) are converted to how many
5316 characters (and bytes) are recorded in members of the structure
5319 If REPLACE is nonzero, we do various things as if the original text
5320 is deleted and a new text is inserted. See the comments in
5321 replace_range (insdel.c) to know what we are doing.
5323 If REPLACE is zero, it is assumed that the source text is unibyte.
5324 Otherwise, it is assumed that the source text is multibyte. */
5327 code_convert_region (from
, from_byte
, to
, to_byte
, coding
, encodep
, replace
)
5328 int from
, from_byte
, to
, to_byte
, encodep
, replace
;
5329 struct coding_system
*coding
;
5331 int len
= to
- from
, len_byte
= to_byte
- from_byte
;
5332 int nchars_del
= 0, nbytes_del
= 0;
5333 int require
, inserted
, inserted_byte
;
5334 int head_skip
, tail_skip
, total_skip
= 0;
5335 Lisp_Object saved_coding_symbol
;
5337 unsigned char *src
, *dst
;
5338 Lisp_Object deletion
;
5339 int orig_point
= PT
, orig_len
= len
;
5341 int multibyte_p
= !NILP (current_buffer
->enable_multibyte_characters
);
5344 saved_coding_symbol
= coding
->symbol
;
5346 if (from
< PT
&& PT
< to
)
5348 TEMP_SET_PT_BOTH (from
, from_byte
);
5354 int saved_from
= from
;
5355 int saved_inhibit_modification_hooks
;
5357 prepare_to_modify_buffer (from
, to
, &from
);
5358 if (saved_from
!= from
)
5361 from_byte
= CHAR_TO_BYTE (from
), to_byte
= CHAR_TO_BYTE (to
);
5362 len_byte
= to_byte
- from_byte
;
5365 /* The code conversion routine can not preserve text properties
5366 for now. So, we must remove all text properties in the
5367 region. Here, we must suppress all modification hooks. */
5368 saved_inhibit_modification_hooks
= inhibit_modification_hooks
;
5369 inhibit_modification_hooks
= 1;
5370 Fset_text_properties (make_number (from
), make_number (to
), Qnil
, Qnil
);
5371 inhibit_modification_hooks
= saved_inhibit_modification_hooks
;
5374 if (! encodep
&& CODING_REQUIRE_DETECTION (coding
))
5376 /* We must detect encoding of text and eol format. */
5378 if (from
< GPT
&& to
> GPT
)
5379 move_gap_both (from
, from_byte
);
5380 if (coding
->type
== coding_type_undecided
)
5382 detect_coding (coding
, BYTE_POS_ADDR (from_byte
), len_byte
);
5383 if (coding
->type
== coding_type_undecided
)
5385 /* It seems that the text contains only ASCII, but we
5386 should not leave it undecided because the deeper
5387 decoding routine (decode_coding) tries to detect the
5388 encodings again in vain. */
5389 coding
->type
= coding_type_emacs_mule
;
5390 coding
->category_idx
= CODING_CATEGORY_IDX_EMACS_MULE
;
5391 /* As emacs-mule decoder will handle composition, we
5392 need this setting to allocate coding->cmp_data
5394 coding
->composing
= COMPOSITION_NO
;
5397 if (coding
->eol_type
== CODING_EOL_UNDECIDED
5398 && coding
->type
!= coding_type_ccl
)
5400 detect_eol (coding
, BYTE_POS_ADDR (from_byte
), len_byte
);
5401 if (coding
->eol_type
== CODING_EOL_UNDECIDED
)
5402 coding
->eol_type
= CODING_EOL_LF
;
5403 /* We had better recover the original eol format if we
5404 encounter an inconsistent eol format while decoding. */
5405 coding
->mode
|= CODING_MODE_INHIBIT_INCONSISTENT_EOL
;
5409 /* Now we convert the text. */
5411 /* For encoding, we must process pre-write-conversion in advance. */
5412 if (! inhibit_pre_post_conversion
5414 && SYMBOLP (coding
->pre_write_conversion
)
5415 && ! NILP (Ffboundp (coding
->pre_write_conversion
)))
5417 /* The function in pre-write-conversion may put a new text in a
5419 struct buffer
*prev
= current_buffer
;
5422 record_unwind_protect (code_convert_region_unwind
, Qnil
);
5423 /* We should not call any more pre-write/post-read-conversion
5424 functions while this pre-write-conversion is running. */
5425 inhibit_pre_post_conversion
= 1;
5426 call2 (coding
->pre_write_conversion
,
5427 make_number (from
), make_number (to
));
5428 inhibit_pre_post_conversion
= 0;
5429 /* Discard the unwind protect. */
5432 if (current_buffer
!= prev
)
5435 new = Fcurrent_buffer ();
5436 set_buffer_internal_1 (prev
);
5437 del_range_2 (from
, from_byte
, to
, to_byte
, 0);
5438 TEMP_SET_PT_BOTH (from
, from_byte
);
5439 insert_from_buffer (XBUFFER (new), 1, len
, 0);
5441 if (orig_point
>= to
)
5442 orig_point
+= len
- orig_len
;
5443 else if (orig_point
> from
)
5447 from_byte
= CHAR_TO_BYTE (from
);
5448 to_byte
= CHAR_TO_BYTE (to
);
5449 len_byte
= to_byte
- from_byte
;
5450 TEMP_SET_PT_BOTH (from
, from_byte
);
5456 if (! EQ (current_buffer
->undo_list
, Qt
))
5457 deletion
= make_buffer_string_both (from
, from_byte
, to
, to_byte
, 1);
5460 nchars_del
= to
- from
;
5461 nbytes_del
= to_byte
- from_byte
;
5465 if (coding
->composing
!= COMPOSITION_DISABLED
)
5468 coding_save_composition (coding
, from
, to
, Fcurrent_buffer ());
5470 coding_allocate_composition_data (coding
, from
);
5473 /* Try to skip the heading and tailing ASCIIs. */
5474 if (coding
->type
!= coding_type_ccl
)
5476 int from_byte_orig
= from_byte
, to_byte_orig
= to_byte
;
5478 if (from
< GPT
&& GPT
< to
)
5479 move_gap_both (from
, from_byte
);
5480 SHRINK_CONVERSION_REGION (&from_byte
, &to_byte
, coding
, NULL
, encodep
);
5481 if (from_byte
== to_byte
5482 && (encodep
|| NILP (coding
->post_read_conversion
))
5483 && ! CODING_REQUIRE_FLUSHING (coding
))
5485 coding
->produced
= len_byte
;
5486 coding
->produced_char
= len
;
5488 /* We must record and adjust for this new text now. */
5489 adjust_after_insert (from
, from_byte_orig
, to
, to_byte_orig
, len
);
5493 head_skip
= from_byte
- from_byte_orig
;
5494 tail_skip
= to_byte_orig
- to_byte
;
5495 total_skip
= head_skip
+ tail_skip
;
5498 len
-= total_skip
; len_byte
-= total_skip
;
5501 /* For conversion, we must put the gap before the text in addition to
5502 making the gap larger for efficient decoding. The required gap
5503 size starts from 2000 which is the magic number used in make_gap.
5504 But, after one batch of conversion, it will be incremented if we
5505 find that it is not enough . */
5508 if (GAP_SIZE
< require
)
5509 make_gap (require
- GAP_SIZE
);
5510 move_gap_both (from
, from_byte
);
5512 inserted
= inserted_byte
= 0;
5514 GAP_SIZE
+= len_byte
;
5517 ZV_BYTE
-= len_byte
;
5520 if (GPT
- BEG
< BEG_UNCHANGED
)
5521 BEG_UNCHANGED
= GPT
- BEG
;
5522 if (Z
- GPT
< END_UNCHANGED
)
5523 END_UNCHANGED
= Z
- GPT
;
5525 if (!encodep
&& coding
->src_multibyte
)
5527 /* Decoding routines expects that the source text is unibyte.
5528 We must convert 8-bit characters of multibyte form to
5530 int len_byte_orig
= len_byte
;
5531 len_byte
= str_as_unibyte (GAP_END_ADDR
- len_byte
, len_byte
);
5532 if (len_byte
< len_byte_orig
)
5533 safe_bcopy (GAP_END_ADDR
- len_byte_orig
, GAP_END_ADDR
- len_byte
,
5535 coding
->src_multibyte
= 0;
5542 /* The buffer memory is now:
5543 +--------+converted-text+---------+-------original-text-------+---+
5544 |<-from->|<--inserted-->|---------|<--------len_byte--------->|---|
5545 |<---------------------- GAP ----------------------->| */
5546 src
= GAP_END_ADDR
- len_byte
;
5547 dst
= GPT_ADDR
+ inserted_byte
;
5550 result
= encode_coding (coding
, src
, dst
, len_byte
, 0);
5553 if (coding
->composing
!= COMPOSITION_DISABLED
)
5554 coding
->cmp_data
->char_offset
= from
+ inserted
;
5555 result
= decode_coding (coding
, src
, dst
, len_byte
, 0);
5558 /* The buffer memory is now:
5559 +--------+-------converted-text----+--+------original-text----+---+
5560 |<-from->|<-inserted->|<-produced->|--|<-(len_byte-consumed)->|---|
5561 |<---------------------- GAP ----------------------->| */
5563 inserted
+= coding
->produced_char
;
5564 inserted_byte
+= coding
->produced
;
5565 len_byte
-= coding
->consumed
;
5567 if (result
== CODING_FINISH_INSUFFICIENT_CMP
)
5569 coding_allocate_composition_data (coding
, from
+ inserted
);
5573 src
+= coding
->consumed
;
5574 dst
+= coding
->produced
;
5576 if (result
== CODING_FINISH_NORMAL
)
5581 if (! encodep
&& result
== CODING_FINISH_INCONSISTENT_EOL
)
5583 unsigned char *pend
= dst
, *p
= pend
- inserted_byte
;
5584 Lisp_Object eol_type
;
5586 /* Encode LFs back to the original eol format (CR or CRLF). */
5587 if (coding
->eol_type
== CODING_EOL_CR
)
5589 while (p
< pend
) if (*p
++ == '\n') p
[-1] = '\r';
5595 while (p
< pend
) if (*p
++ == '\n') count
++;
5596 if (src
- dst
< count
)
5598 /* We don't have sufficient room for encoding LFs
5599 back to CRLF. We must record converted and
5600 not-yet-converted text back to the buffer
5601 content, enlarge the gap, then record them out of
5602 the buffer contents again. */
5603 int add
= len_byte
+ inserted_byte
;
5606 ZV
+= add
; Z
+= add
; ZV_BYTE
+= add
; Z_BYTE
+= add
;
5607 GPT
+= inserted_byte
; GPT_BYTE
+= inserted_byte
;
5608 make_gap (count
- GAP_SIZE
);
5610 ZV
-= add
; Z
-= add
; ZV_BYTE
-= add
; Z_BYTE
-= add
;
5611 GPT
-= inserted_byte
; GPT_BYTE
-= inserted_byte
;
5612 /* Don't forget to update SRC, DST, and PEND. */
5613 src
= GAP_END_ADDR
- len_byte
;
5614 dst
= GPT_ADDR
+ inserted_byte
;
5618 inserted_byte
+= count
;
5619 coding
->produced
+= count
;
5620 p
= dst
= pend
+ count
;
5624 if (*p
== '\n') count
--, *--p
= '\r';
5628 /* Suppress eol-format conversion in the further conversion. */
5629 coding
->eol_type
= CODING_EOL_LF
;
5631 /* Set the coding system symbol to that for Unix-like EOL. */
5632 eol_type
= Fget (saved_coding_symbol
, Qeol_type
);
5633 if (VECTORP (eol_type
)
5634 && XVECTOR (eol_type
)->size
== 3
5635 && SYMBOLP (XVECTOR (eol_type
)->contents
[CODING_EOL_LF
]))
5636 coding
->symbol
= XVECTOR (eol_type
)->contents
[CODING_EOL_LF
];
5638 coding
->symbol
= saved_coding_symbol
;
5644 if (coding
->type
!= coding_type_ccl
5645 || coding
->mode
& CODING_MODE_LAST_BLOCK
)
5647 coding
->mode
|= CODING_MODE_LAST_BLOCK
;
5650 if (result
== CODING_FINISH_INSUFFICIENT_SRC
)
5652 /* The source text ends in invalid codes. Let's just
5653 make them valid buffer contents, and finish conversion. */
5656 unsigned char *start
= dst
;
5658 inserted
+= len_byte
;
5662 dst
+= CHAR_STRING (c
, dst
);
5665 inserted_byte
+= dst
- start
;
5669 inserted
+= len_byte
;
5670 inserted_byte
+= len_byte
;
5676 if (result
== CODING_FINISH_INTERRUPT
)
5678 /* The conversion procedure was interrupted by a user. */
5681 /* Now RESULT == CODING_FINISH_INSUFFICIENT_DST */
5682 if (coding
->consumed
< 1)
5684 /* It's quite strange to require more memory without
5685 consuming any bytes. Perhaps CCL program bug. */
5690 /* We have just done the first batch of conversion which was
5691 stopped because of insufficient gap. Let's reconsider the
5692 required gap size (i.e. SRT - DST) now.
5694 We have converted ORIG bytes (== coding->consumed) into
5695 NEW bytes (coding->produced). To convert the remaining
5696 LEN bytes, we may need REQUIRE bytes of gap, where:
5697 REQUIRE + LEN_BYTE = LEN_BYTE * (NEW / ORIG)
5698 REQUIRE = LEN_BYTE * (NEW - ORIG) / ORIG
5699 Here, we are sure that NEW >= ORIG. */
5700 float ratio
= coding
->produced
- coding
->consumed
;
5701 ratio
/= coding
->consumed
;
5702 require
= len_byte
* ratio
;
5705 if ((src
- dst
) < (require
+ 2000))
5707 /* See the comment above the previous call of make_gap. */
5708 int add
= len_byte
+ inserted_byte
;
5711 ZV
+= add
; Z
+= add
; ZV_BYTE
+= add
; Z_BYTE
+= add
;
5712 GPT
+= inserted_byte
; GPT_BYTE
+= inserted_byte
;
5713 make_gap (require
+ 2000);
5715 ZV
-= add
; Z
-= add
; ZV_BYTE
-= add
; Z_BYTE
-= add
;
5716 GPT
-= inserted_byte
; GPT_BYTE
-= inserted_byte
;
5719 if (src
- dst
> 0) *dst
= 0; /* Put an anchor. */
5721 if (encodep
&& coding
->dst_multibyte
)
5723 /* The output is unibyte. We must convert 8-bit characters to
5725 if (inserted_byte
* 2 > GAP_SIZE
)
5727 GAP_SIZE
-= inserted_byte
;
5728 ZV
+= inserted_byte
; Z
+= inserted_byte
;
5729 ZV_BYTE
+= inserted_byte
; Z_BYTE
+= inserted_byte
;
5730 GPT
+= inserted_byte
; GPT_BYTE
+= inserted_byte
;
5731 make_gap (inserted_byte
- GAP_SIZE
);
5732 GAP_SIZE
+= inserted_byte
;
5733 ZV
-= inserted_byte
; Z
-= inserted_byte
;
5734 ZV_BYTE
-= inserted_byte
; Z_BYTE
-= inserted_byte
;
5735 GPT
-= inserted_byte
; GPT_BYTE
-= inserted_byte
;
5737 inserted_byte
= str_to_multibyte (GPT_ADDR
, GAP_SIZE
, inserted_byte
);
5740 /* If we shrank the conversion area, adjust it now. */
5744 safe_bcopy (GAP_END_ADDR
, GPT_ADDR
+ inserted_byte
, tail_skip
);
5745 inserted
+= total_skip
; inserted_byte
+= total_skip
;
5746 GAP_SIZE
+= total_skip
;
5747 GPT
-= head_skip
; GPT_BYTE
-= head_skip
;
5748 ZV
-= total_skip
; ZV_BYTE
-= total_skip
;
5749 Z
-= total_skip
; Z_BYTE
-= total_skip
;
5750 from
-= head_skip
; from_byte
-= head_skip
;
5751 to
+= tail_skip
; to_byte
+= tail_skip
;
5755 if (! EQ (current_buffer
->undo_list
, Qt
))
5756 adjust_after_replace (from
, from_byte
, deletion
, inserted
, inserted_byte
);
5758 adjust_after_replace_noundo (from
, from_byte
, nchars_del
, nbytes_del
,
5759 inserted
, inserted_byte
);
5760 inserted
= Z
- prev_Z
;
5762 if (!encodep
&& coding
->cmp_data
&& coding
->cmp_data
->used
)
5763 coding_restore_composition (coding
, Fcurrent_buffer ());
5764 coding_free_composition_data (coding
);
5766 if (! inhibit_pre_post_conversion
5767 && ! encodep
&& ! NILP (coding
->post_read_conversion
))
5772 TEMP_SET_PT_BOTH (from
, from_byte
);
5774 record_unwind_protect (code_convert_region_unwind
, Qnil
);
5775 /* We should not call any more pre-write/post-read-conversion
5776 functions while this post-read-conversion is running. */
5777 inhibit_pre_post_conversion
= 1;
5778 val
= call1 (coding
->post_read_conversion
, make_number (inserted
));
5779 inhibit_pre_post_conversion
= 0;
5780 /* Discard the unwind protect. */
5783 inserted
+= Z
- prev_Z
;
5786 if (orig_point
>= from
)
5788 if (orig_point
>= from
+ orig_len
)
5789 orig_point
+= inserted
- orig_len
;
5792 TEMP_SET_PT (orig_point
);
5797 signal_after_change (from
, to
- from
, inserted
);
5798 update_compositions (from
, from
+ inserted
, CHECK_BORDER
);
5802 coding
->consumed
= to_byte
- from_byte
;
5803 coding
->consumed_char
= to
- from
;
5804 coding
->produced
= inserted_byte
;
5805 coding
->produced_char
= inserted
;
5812 run_pre_post_conversion_on_str (str
, coding
, encodep
)
5814 struct coding_system
*coding
;
5817 int count
= SPECPDL_INDEX ();
5818 struct gcpro gcpro1
, gcpro2
;
5819 int multibyte
= STRING_MULTIBYTE (str
);
5822 Lisp_Object old_deactivate_mark
;
5824 record_unwind_protect (Fset_buffer
, Fcurrent_buffer ());
5825 record_unwind_protect (code_convert_region_unwind
, Qnil
);
5826 /* It is not crucial to specbind this. */
5827 old_deactivate_mark
= Vdeactivate_mark
;
5828 GCPRO2 (str
, old_deactivate_mark
);
5830 buffer
= Fget_buffer_create (build_string (" *code-converting-work*"));
5831 buf
= XBUFFER (buffer
);
5833 buf
->directory
= current_buffer
->directory
;
5834 buf
->read_only
= Qnil
;
5835 buf
->filename
= Qnil
;
5836 buf
->undo_list
= Qt
;
5837 buf
->overlays_before
= Qnil
;
5838 buf
->overlays_after
= Qnil
;
5840 set_buffer_internal (buf
);
5841 /* We must insert the contents of STR as is without
5842 unibyte<->multibyte conversion. For that, we adjust the
5843 multibyteness of the working buffer to that of STR. */
5845 buf
->enable_multibyte_characters
= multibyte
? Qt
: Qnil
;
5847 insert_from_string (str
, 0, 0,
5848 SCHARS (str
), SBYTES (str
), 0);
5850 inhibit_pre_post_conversion
= 1;
5852 call2 (coding
->pre_write_conversion
, make_number (BEG
), make_number (Z
));
5855 TEMP_SET_PT_BOTH (BEG
, BEG_BYTE
);
5856 call1 (coding
->post_read_conversion
, make_number (Z
- BEG
));
5858 inhibit_pre_post_conversion
= 0;
5859 Vdeactivate_mark
= old_deactivate_mark
;
5860 str
= make_buffer_string (BEG
, Z
, 1);
5861 return unbind_to (count
, str
);
5865 decode_coding_string (str
, coding
, nocopy
)
5867 struct coding_system
*coding
;
5871 struct conversion_buffer buf
;
5873 Lisp_Object saved_coding_symbol
;
5875 int require_decoding
;
5876 int shrinked_bytes
= 0;
5878 int consumed
, consumed_char
, produced
, produced_char
;
5881 to_byte
= SBYTES (str
);
5883 saved_coding_symbol
= coding
->symbol
;
5884 coding
->src_multibyte
= STRING_MULTIBYTE (str
);
5885 coding
->dst_multibyte
= 1;
5886 if (CODING_REQUIRE_DETECTION (coding
))
5888 /* See the comments in code_convert_region. */
5889 if (coding
->type
== coding_type_undecided
)
5891 detect_coding (coding
, SDATA (str
), to_byte
);
5892 if (coding
->type
== coding_type_undecided
)
5894 coding
->type
= coding_type_emacs_mule
;
5895 coding
->category_idx
= CODING_CATEGORY_IDX_EMACS_MULE
;
5896 /* As emacs-mule decoder will handle composition, we
5897 need this setting to allocate coding->cmp_data
5899 coding
->composing
= COMPOSITION_NO
;
5902 if (coding
->eol_type
== CODING_EOL_UNDECIDED
5903 && coding
->type
!= coding_type_ccl
)
5905 saved_coding_symbol
= coding
->symbol
;
5906 detect_eol (coding
, SDATA (str
), to_byte
);
5907 if (coding
->eol_type
== CODING_EOL_UNDECIDED
)
5908 coding
->eol_type
= CODING_EOL_LF
;
5909 /* We had better recover the original eol format if we
5910 encounter an inconsistent eol format while decoding. */
5911 coding
->mode
|= CODING_MODE_INHIBIT_INCONSISTENT_EOL
;
5915 if (coding
->type
== coding_type_no_conversion
5916 || coding
->type
== coding_type_raw_text
)
5917 coding
->dst_multibyte
= 0;
5919 require_decoding
= CODING_REQUIRE_DECODING (coding
);
5921 if (STRING_MULTIBYTE (str
))
5923 /* Decoding routines expect the source text to be unibyte. */
5924 str
= Fstring_as_unibyte (str
);
5925 to_byte
= SBYTES (str
);
5927 coding
->src_multibyte
= 0;
5930 /* Try to skip the heading and tailing ASCIIs. */
5931 if (require_decoding
&& coding
->type
!= coding_type_ccl
)
5933 SHRINK_CONVERSION_REGION (&from
, &to_byte
, coding
, SDATA (str
),
5935 if (from
== to_byte
)
5936 require_decoding
= 0;
5937 shrinked_bytes
= from
+ (SBYTES (str
) - to_byte
);
5940 if (!require_decoding
)
5942 coding
->consumed
= SBYTES (str
);
5943 coding
->consumed_char
= SCHARS (str
);
5944 if (coding
->dst_multibyte
)
5946 str
= Fstring_as_multibyte (str
);
5949 coding
->produced
= SBYTES (str
);
5950 coding
->produced_char
= SCHARS (str
);
5951 return (nocopy
? str
: Fcopy_sequence (str
));
5954 if (coding
->composing
!= COMPOSITION_DISABLED
)
5955 coding_allocate_composition_data (coding
, from
);
5956 len
= decoding_buffer_size (coding
, to_byte
- from
);
5957 allocate_conversion_buffer (buf
, len
);
5959 consumed
= consumed_char
= produced
= produced_char
= 0;
5962 result
= decode_coding (coding
, SDATA (str
) + from
+ consumed
,
5963 buf
.data
+ produced
, to_byte
- from
- consumed
,
5964 buf
.size
- produced
);
5965 consumed
+= coding
->consumed
;
5966 consumed_char
+= coding
->consumed_char
;
5967 produced
+= coding
->produced
;
5968 produced_char
+= coding
->produced_char
;
5969 if (result
== CODING_FINISH_NORMAL
5970 || (result
== CODING_FINISH_INSUFFICIENT_SRC
5971 && coding
->consumed
== 0))
5973 if (result
== CODING_FINISH_INSUFFICIENT_CMP
)
5974 coding_allocate_composition_data (coding
, from
+ produced_char
);
5975 else if (result
== CODING_FINISH_INSUFFICIENT_DST
)
5976 extend_conversion_buffer (&buf
);
5977 else if (result
== CODING_FINISH_INCONSISTENT_EOL
)
5979 Lisp_Object eol_type
;
5981 /* Recover the original EOL format. */
5982 if (coding
->eol_type
== CODING_EOL_CR
)
5985 for (p
= buf
.data
; p
< buf
.data
+ produced
; p
++)
5986 if (*p
== '\n') *p
= '\r';
5988 else if (coding
->eol_type
== CODING_EOL_CRLF
)
5991 unsigned char *p0
, *p1
;
5992 for (p0
= buf
.data
, p1
= p0
+ produced
; p0
< p1
; p0
++)
5993 if (*p0
== '\n') num_eol
++;
5994 if (produced
+ num_eol
>= buf
.size
)
5995 extend_conversion_buffer (&buf
);
5996 for (p0
= buf
.data
+ produced
, p1
= p0
+ num_eol
; p0
> buf
.data
;)
5999 if (*p0
== '\n') *--p1
= '\r';
6001 produced
+= num_eol
;
6002 produced_char
+= num_eol
;
6004 /* Suppress eol-format conversion in the further conversion. */
6005 coding
->eol_type
= CODING_EOL_LF
;
6007 /* Set the coding system symbol to that for Unix-like EOL. */
6008 eol_type
= Fget (saved_coding_symbol
, Qeol_type
);
6009 if (VECTORP (eol_type
)
6010 && XVECTOR (eol_type
)->size
== 3
6011 && SYMBOLP (XVECTOR (eol_type
)->contents
[CODING_EOL_LF
]))
6012 coding
->symbol
= XVECTOR (eol_type
)->contents
[CODING_EOL_LF
];
6014 coding
->symbol
= saved_coding_symbol
;
6020 coding
->consumed
= consumed
;
6021 coding
->consumed_char
= consumed_char
;
6022 coding
->produced
= produced
;
6023 coding
->produced_char
= produced_char
;
6025 if (coding
->dst_multibyte
)
6026 newstr
= make_uninit_multibyte_string (produced_char
+ shrinked_bytes
,
6027 produced
+ shrinked_bytes
);
6029 newstr
= make_uninit_string (produced
+ shrinked_bytes
);
6031 STRING_COPYIN (newstr
, 0, SDATA (str
), from
);
6032 STRING_COPYIN (newstr
, from
, buf
.data
, produced
);
6033 if (shrinked_bytes
> from
)
6034 STRING_COPYIN (newstr
, from
+ produced
,
6035 SDATA (str
) + to_byte
,
6036 shrinked_bytes
- from
);
6037 free_conversion_buffer (&buf
);
6039 if (coding
->cmp_data
&& coding
->cmp_data
->used
)
6040 coding_restore_composition (coding
, newstr
);
6041 coding_free_composition_data (coding
);
6043 if (SYMBOLP (coding
->post_read_conversion
)
6044 && !NILP (Ffboundp (coding
->post_read_conversion
)))
6045 newstr
= run_pre_post_conversion_on_str (newstr
, coding
, 0);
6051 encode_coding_string (str
, coding
, nocopy
)
6053 struct coding_system
*coding
;
6057 struct conversion_buffer buf
;
6058 int from
, to
, to_byte
;
6060 int shrinked_bytes
= 0;
6062 int consumed
, consumed_char
, produced
, produced_char
;
6064 if (SYMBOLP (coding
->pre_write_conversion
)
6065 && !NILP (Ffboundp (coding
->pre_write_conversion
)))
6066 str
= run_pre_post_conversion_on_str (str
, coding
, 1);
6070 to_byte
= SBYTES (str
);
6072 /* Encoding routines determine the multibyteness of the source text
6073 by coding->src_multibyte. */
6074 coding
->src_multibyte
= STRING_MULTIBYTE (str
);
6075 coding
->dst_multibyte
= 0;
6076 if (! CODING_REQUIRE_ENCODING (coding
))
6078 coding
->consumed
= SBYTES (str
);
6079 coding
->consumed_char
= SCHARS (str
);
6080 if (STRING_MULTIBYTE (str
))
6082 str
= Fstring_as_unibyte (str
);
6085 coding
->produced
= SBYTES (str
);
6086 coding
->produced_char
= SCHARS (str
);
6087 return (nocopy
? str
: Fcopy_sequence (str
));
6090 if (coding
->composing
!= COMPOSITION_DISABLED
)
6091 coding_save_composition (coding
, from
, to
, str
);
6093 /* Try to skip the heading and tailing ASCIIs. */
6094 if (coding
->type
!= coding_type_ccl
)
6096 SHRINK_CONVERSION_REGION (&from
, &to_byte
, coding
, SDATA (str
),
6098 if (from
== to_byte
)
6099 return (nocopy
? str
: Fcopy_sequence (str
));
6100 shrinked_bytes
= from
+ (SBYTES (str
) - to_byte
);
6103 len
= encoding_buffer_size (coding
, to_byte
- from
);
6104 allocate_conversion_buffer (buf
, len
);
6106 consumed
= consumed_char
= produced
= produced_char
= 0;
6109 result
= encode_coding (coding
, SDATA (str
) + from
+ consumed
,
6110 buf
.data
+ produced
, to_byte
- from
- consumed
,
6111 buf
.size
- produced
);
6112 consumed
+= coding
->consumed
;
6113 consumed_char
+= coding
->consumed_char
;
6114 produced
+= coding
->produced
;
6115 produced_char
+= coding
->produced_char
;
6116 if (result
== CODING_FINISH_NORMAL
6117 || (result
== CODING_FINISH_INSUFFICIENT_SRC
6118 && coding
->consumed
== 0))
6120 /* Now result should be CODING_FINISH_INSUFFICIENT_DST. */
6121 extend_conversion_buffer (&buf
);
6124 coding
->consumed
= consumed
;
6125 coding
->consumed_char
= consumed_char
;
6126 coding
->produced
= produced
;
6127 coding
->produced_char
= produced_char
;
6129 newstr
= make_uninit_string (produced
+ shrinked_bytes
);
6131 STRING_COPYIN (newstr
, 0, SDATA (str
), from
);
6132 STRING_COPYIN (newstr
, from
, buf
.data
, produced
);
6133 if (shrinked_bytes
> from
)
6134 STRING_COPYIN (newstr
, from
+ produced
,
6135 SDATA (str
) + to_byte
,
6136 shrinked_bytes
- from
);
6138 free_conversion_buffer (&buf
);
6139 coding_free_composition_data (coding
);
6146 /*** 8. Emacs Lisp library functions ***/
6148 DEFUN ("coding-system-p", Fcoding_system_p
, Scoding_system_p
, 1, 1, 0,
6149 doc
: /* Return t if OBJECT is nil or a coding-system.
6150 See the documentation of `make-coding-system' for information
6151 about coding-system objects. */)
6159 /* Get coding-spec vector for OBJ. */
6160 obj
= Fget (obj
, Qcoding_system
);
6161 return ((VECTORP (obj
) && XVECTOR (obj
)->size
== 5)
6165 DEFUN ("read-non-nil-coding-system", Fread_non_nil_coding_system
,
6166 Sread_non_nil_coding_system
, 1, 1, 0,
6167 doc
: /* Read a coding system from the minibuffer, prompting with string PROMPT. */)
6174 val
= Fcompleting_read (prompt
, Vcoding_system_alist
, Qnil
,
6175 Qt
, Qnil
, Qcoding_system_history
, Qnil
, Qnil
);
6177 while (SCHARS (val
) == 0);
6178 return (Fintern (val
, Qnil
));
6181 DEFUN ("read-coding-system", Fread_coding_system
, Sread_coding_system
, 1, 2, 0,
6182 doc
: /* Read a coding system from the minibuffer, prompting with string PROMPT.
6183 If the user enters null input, return second argument DEFAULT-CODING-SYSTEM. */)
6184 (prompt
, default_coding_system
)
6185 Lisp_Object prompt
, default_coding_system
;
6188 if (SYMBOLP (default_coding_system
))
6189 default_coding_system
= SYMBOL_NAME (default_coding_system
);
6190 val
= Fcompleting_read (prompt
, Vcoding_system_alist
, Qnil
,
6191 Qt
, Qnil
, Qcoding_system_history
,
6192 default_coding_system
, Qnil
);
6193 return (SCHARS (val
) == 0 ? Qnil
: Fintern (val
, Qnil
));
6196 DEFUN ("check-coding-system", Fcheck_coding_system
, Scheck_coding_system
,
6198 doc
: /* Check validity of CODING-SYSTEM.
6199 If valid, return CODING-SYSTEM, else signal a `coding-system-error' error.
6200 It is valid if it is a symbol with a non-nil `coding-system' property.
6201 The value of property should be a vector of length 5. */)
6203 Lisp_Object coding_system
;
6205 CHECK_SYMBOL (coding_system
);
6206 if (!NILP (Fcoding_system_p (coding_system
)))
6207 return coding_system
;
6209 Fsignal (Qcoding_system_error
, Fcons (coding_system
, Qnil
));
6213 detect_coding_system (src
, src_bytes
, highest
, multibytep
)
6214 const unsigned char *src
;
6215 int src_bytes
, highest
;
6218 int coding_mask
, eol_type
;
6219 Lisp_Object val
, tmp
;
6222 coding_mask
= detect_coding_mask (src
, src_bytes
, NULL
, &dummy
, multibytep
);
6223 eol_type
= detect_eol_type (src
, src_bytes
, &dummy
);
6224 if (eol_type
== CODING_EOL_INCONSISTENT
)
6225 eol_type
= CODING_EOL_UNDECIDED
;
6230 if (eol_type
!= CODING_EOL_UNDECIDED
)
6233 val2
= Fget (Qundecided
, Qeol_type
);
6235 val
= XVECTOR (val2
)->contents
[eol_type
];
6237 return (highest
? val
: Fcons (val
, Qnil
));
6240 /* At first, gather possible coding systems in VAL. */
6242 for (tmp
= Vcoding_category_list
; CONSP (tmp
); tmp
= XCDR (tmp
))
6244 Lisp_Object category_val
, category_index
;
6246 category_index
= Fget (XCAR (tmp
), Qcoding_category_index
);
6247 category_val
= Fsymbol_value (XCAR (tmp
));
6248 if (!NILP (category_val
)
6249 && NATNUMP (category_index
)
6250 && (coding_mask
& (1 << XFASTINT (category_index
))))
6252 val
= Fcons (category_val
, val
);
6258 val
= Fnreverse (val
);
6260 /* Then, replace the elements with subsidiary coding systems. */
6261 for (tmp
= val
; CONSP (tmp
); tmp
= XCDR (tmp
))
6263 if (eol_type
!= CODING_EOL_UNDECIDED
6264 && eol_type
!= CODING_EOL_INCONSISTENT
)
6267 eol
= Fget (XCAR (tmp
), Qeol_type
);
6269 XSETCAR (tmp
, XVECTOR (eol
)->contents
[eol_type
]);
6272 return (highest
? XCAR (val
) : val
);
6275 DEFUN ("detect-coding-region", Fdetect_coding_region
, Sdetect_coding_region
,
6277 doc
: /* Detect coding system of the text in the region between START and END.
6278 Return a list of possible coding systems ordered by priority.
6280 If only ASCII characters are found, it returns a list of single element
6281 `undecided' or its subsidiary coding system according to a detected
6284 If optional argument HIGHEST is non-nil, return the coding system of
6285 highest priority. */)
6286 (start
, end
, highest
)
6287 Lisp_Object start
, end
, highest
;
6290 int from_byte
, to_byte
;
6291 int include_anchor_byte
= 0;
6293 CHECK_NUMBER_COERCE_MARKER (start
);
6294 CHECK_NUMBER_COERCE_MARKER (end
);
6296 validate_region (&start
, &end
);
6297 from
= XINT (start
), to
= XINT (end
);
6298 from_byte
= CHAR_TO_BYTE (from
);
6299 to_byte
= CHAR_TO_BYTE (to
);
6301 if (from
< GPT
&& to
>= GPT
)
6302 move_gap_both (to
, to_byte
);
6303 /* If we an anchor byte `\0' follows the region, we include it in
6304 the detecting source. Then code detectors can handle the tailing
6305 byte sequence more accurately.
6307 Fix me: This is not an perfect solution. It is better that we
6308 add one more argument, say LAST_BLOCK, to all detect_coding_XXX.
6310 if (to
== Z
|| (to
== GPT
&& GAP_SIZE
> 0))
6311 include_anchor_byte
= 1;
6312 return detect_coding_system (BYTE_POS_ADDR (from_byte
),
6313 to_byte
- from_byte
+ include_anchor_byte
,
6315 !NILP (current_buffer
6316 ->enable_multibyte_characters
));
6319 DEFUN ("detect-coding-string", Fdetect_coding_string
, Sdetect_coding_string
,
6321 doc
: /* Detect coding system of the text in STRING.
6322 Return a list of possible coding systems ordered by priority.
6324 If only ASCII characters are found, it returns a list of single element
6325 `undecided' or its subsidiary coding system according to a detected
6328 If optional argument HIGHEST is non-nil, return the coding system of
6329 highest priority. */)
6331 Lisp_Object string
, highest
;
6333 CHECK_STRING (string
);
6335 return detect_coding_system (SDATA (string
),
6336 /* "+ 1" is to include the anchor byte
6337 `\0'. With this, code detectors can
6338 handle the tailing bytes more
6340 SBYTES (string
) + 1,
6342 STRING_MULTIBYTE (string
));
6345 /* Return an intersection of lists L1 and L2. */
6348 intersection (l1
, l2
)
6351 Lisp_Object val
= Fcons (Qnil
, Qnil
), tail
;
6353 for (tail
= val
; CONSP (l1
); l1
= XCDR (l1
))
6355 if (!NILP (Fmemq (XCAR (l1
), l2
)))
6357 XSETCDR (tail
, Fcons (XCAR (l1
), Qnil
));
6365 /* Subroutine for Fsafe_coding_systems_region_internal.
6367 Return a list of coding systems that safely encode the multibyte
6368 text between P and PEND. SAFE_CODINGS, if non-nil, is a list of
6369 possible coding systems. If it is nil, it means that we have not
6370 yet found any coding systems.
6372 WORK_TABLE is a copy of the char-table Vchar_coding_system_table. An
6373 element of WORK_TABLE is set to t once the element is looked up.
6375 If a non-ASCII single byte char is found, set
6376 *single_byte_char_found to 1. */
6379 find_safe_codings (p
, pend
, safe_codings
, work_table
, single_byte_char_found
)
6380 unsigned char *p
, *pend
;
6381 Lisp_Object safe_codings
, work_table
;
6382 int *single_byte_char_found
;
6389 c
= STRING_CHAR_AND_LENGTH (p
, pend
- p
, len
);
6391 if (ASCII_BYTE_P (c
))
6392 /* We can ignore ASCII characters here. */
6394 if (SINGLE_BYTE_CHAR_P (c
))
6395 *single_byte_char_found
= 1;
6396 if (NILP (safe_codings
))
6398 /* Check the safe coding systems for C. */
6399 val
= char_table_ref_and_index (work_table
, c
, &idx
);
6401 /* This element was already checked. Ignore it. */
6403 /* Remember that we checked this element. */
6404 CHAR_TABLE_SET (work_table
, make_number (idx
), Qt
);
6406 /* If there are some safe coding systems for C and we have
6407 already found the other set of coding systems for the
6408 different characters, get the intersection of them. */
6409 if (!EQ (safe_codings
, Qt
) && !NILP (val
))
6410 val
= intersection (safe_codings
, val
);
6413 return safe_codings
;
6417 /* Return a list of coding systems that safely encode the text between
6418 START and END. If the text contains only ASCII or is unibyte,
6421 DEFUN ("find-coding-systems-region-internal",
6422 Ffind_coding_systems_region_internal
,
6423 Sfind_coding_systems_region_internal
, 2, 2, 0,
6424 doc
: /* Internal use only. */)
6426 Lisp_Object start
, end
;
6428 Lisp_Object work_table
, safe_codings
;
6429 int non_ascii_p
= 0;
6430 int single_byte_char_found
= 0;
6431 const unsigned char *p1
, *p1end
, *p2
, *p2end
, *p
;
6433 if (STRINGP (start
))
6435 if (!STRING_MULTIBYTE (start
))
6437 p1
= SDATA (start
), p1end
= p1
+ SBYTES (start
);
6439 if (SCHARS (start
) != SBYTES (start
))
6446 CHECK_NUMBER_COERCE_MARKER (start
);
6447 CHECK_NUMBER_COERCE_MARKER (end
);
6448 if (XINT (start
) < BEG
|| XINT (end
) > Z
|| XINT (start
) > XINT (end
))
6449 args_out_of_range (start
, end
);
6450 if (NILP (current_buffer
->enable_multibyte_characters
))
6452 from
= CHAR_TO_BYTE (XINT (start
));
6453 to
= CHAR_TO_BYTE (XINT (end
));
6454 stop
= from
< GPT_BYTE
&& GPT_BYTE
< to
? GPT_BYTE
: to
;
6455 p1
= BYTE_POS_ADDR (from
), p1end
= p1
+ (stop
- from
);
6459 p2
= BYTE_POS_ADDR (stop
), p2end
= p2
+ (to
- stop
);
6460 if (XINT (end
) - XINT (start
) != to
- from
)
6466 /* We are sure that the text contains no multibyte character.
6467 Check if it contains eight-bit-graphic. */
6469 for (p
= p1
; p
< p1end
&& ASCII_BYTE_P (*p
); p
++);
6472 for (p
= p2
; p
< p2end
&& ASCII_BYTE_P (*p
); p
++);
6478 /* The text contains non-ASCII characters. */
6479 work_table
= Fcopy_sequence (Vchar_coding_system_table
);
6480 safe_codings
= find_safe_codings (p1
, p1end
, Qt
, work_table
,
6481 &single_byte_char_found
);
6483 safe_codings
= find_safe_codings (p2
, p2end
, safe_codings
, work_table
,
6484 &single_byte_char_found
);
6486 if (EQ (safe_codings
, Qt
))
6487 ; /* Nothing to be done. */
6488 else if (!single_byte_char_found
)
6490 /* Append generic coding systems. */
6491 Lisp_Object args
[2];
6492 args
[0] = safe_codings
;
6493 args
[1] = Fchar_table_extra_slot (Vchar_coding_system_table
,
6495 safe_codings
= Fappend (2, args
);
6498 safe_codings
= Fcons (Qraw_text
,
6500 Fcons (Qno_conversion
, safe_codings
)));
6501 return safe_codings
;
6505 /* Search from position POS for such characters that are unencodable
6506 accoding to SAFE_CHARS, and return a list of their positions. P
6507 points where in the memory the character at POS exists. Limit the
6508 search at PEND or when Nth unencodable characters are found.
6510 If SAFE_CHARS is a char table, an element for an unencodable
6513 If SAFE_CHARS is nil, all non-ASCII characters are unencodable.
6515 Otherwise, SAFE_CHARS is t, and only eight-bit-contrl and
6516 eight-bit-graphic characters are unencodable. */
6519 unencodable_char_position (safe_chars
, pos
, p
, pend
, n
)
6520 Lisp_Object safe_chars
;
6522 unsigned char *p
, *pend
;
6525 Lisp_Object pos_list
;
6531 int c
= STRING_CHAR_AND_LENGTH (p
, MAX_MULTIBYTE_LENGTH
, len
);
6534 && (CHAR_TABLE_P (safe_chars
)
6535 ? NILP (CHAR_TABLE_REF (safe_chars
, c
))
6536 : (NILP (safe_chars
) || c
< 256)))
6538 pos_list
= Fcons (make_number (pos
), pos_list
);
6545 return Fnreverse (pos_list
);
6549 DEFUN ("unencodable-char-position", Funencodable_char_position
,
6550 Sunencodable_char_position
, 3, 5, 0,
6552 Return position of first un-encodable character in a region.
6553 START and END specfiy the region and CODING-SYSTEM specifies the
6554 encoding to check. Return nil if CODING-SYSTEM does encode the region.
6556 If optional 4th argument COUNT is non-nil, it specifies at most how
6557 many un-encodable characters to search. In this case, the value is a
6560 If optional 5th argument STRING is non-nil, it is a string to search
6561 for un-encodable characters. In that case, START and END are indexes
6563 (start
, end
, coding_system
, count
, string
)
6564 Lisp_Object start
, end
, coding_system
, count
, string
;
6567 Lisp_Object safe_chars
;
6568 struct coding_system coding
;
6569 Lisp_Object positions
;
6571 unsigned char *p
, *pend
;
6575 validate_region (&start
, &end
);
6576 from
= XINT (start
);
6578 if (NILP (current_buffer
->enable_multibyte_characters
))
6580 p
= CHAR_POS_ADDR (from
);
6581 pend
= CHAR_POS_ADDR (to
);
6585 CHECK_STRING (string
);
6586 CHECK_NATNUM (start
);
6588 from
= XINT (start
);
6591 || to
> SCHARS (string
))
6592 args_out_of_range_3 (string
, start
, end
);
6593 if (! STRING_MULTIBYTE (string
))
6595 p
= SDATA (string
) + string_char_to_byte (string
, from
);
6596 pend
= SDATA (string
) + string_char_to_byte (string
, to
);
6599 setup_coding_system (Fcheck_coding_system (coding_system
), &coding
);
6605 CHECK_NATNUM (count
);
6609 if (coding
.type
== coding_type_no_conversion
6610 || coding
.type
== coding_type_raw_text
)
6613 if (coding
.type
== coding_type_undecided
)
6616 safe_chars
= coding_safe_chars (&coding
);
6618 if (STRINGP (string
)
6619 || from
>= GPT
|| to
<= GPT
)
6620 positions
= unencodable_char_position (safe_chars
, from
, p
, pend
, n
);
6623 Lisp_Object args
[2];
6625 args
[0] = unencodable_char_position (safe_chars
, from
, p
, GPT_ADDR
, n
);
6626 n
-= XINT (Flength (args
[0]));
6628 positions
= args
[0];
6631 args
[1] = unencodable_char_position (safe_chars
, GPT
, GAP_END_ADDR
,
6633 positions
= Fappend (2, args
);
6637 return (NILP (count
) ? Fcar (positions
) : positions
);
6642 code_convert_region1 (start
, end
, coding_system
, encodep
)
6643 Lisp_Object start
, end
, coding_system
;
6646 struct coding_system coding
;
6649 CHECK_NUMBER_COERCE_MARKER (start
);
6650 CHECK_NUMBER_COERCE_MARKER (end
);
6651 CHECK_SYMBOL (coding_system
);
6653 validate_region (&start
, &end
);
6654 from
= XFASTINT (start
);
6655 to
= XFASTINT (end
);
6657 if (NILP (coding_system
))
6658 return make_number (to
- from
);
6660 if (setup_coding_system (Fcheck_coding_system (coding_system
), &coding
) < 0)
6661 error ("Invalid coding system: %s", SDATA (SYMBOL_NAME (coding_system
)));
6663 coding
.mode
|= CODING_MODE_LAST_BLOCK
;
6664 coding
.src_multibyte
= coding
.dst_multibyte
6665 = !NILP (current_buffer
->enable_multibyte_characters
);
6666 code_convert_region (from
, CHAR_TO_BYTE (from
), to
, CHAR_TO_BYTE (to
),
6667 &coding
, encodep
, 1);
6668 Vlast_coding_system_used
= coding
.symbol
;
6669 return make_number (coding
.produced_char
);
6672 DEFUN ("decode-coding-region", Fdecode_coding_region
, Sdecode_coding_region
,
6673 3, 3, "r\nzCoding system: ",
6674 doc
: /* Decode the current region from the specified coding system.
6675 When called from a program, takes three arguments:
6676 START, END, and CODING-SYSTEM. START and END are buffer positions.
6677 This function sets `last-coding-system-used' to the precise coding system
6678 used (which may be different from CODING-SYSTEM if CODING-SYSTEM is
6679 not fully specified.)
6680 It returns the length of the decoded text. */)
6681 (start
, end
, coding_system
)
6682 Lisp_Object start
, end
, coding_system
;
6684 return code_convert_region1 (start
, end
, coding_system
, 0);
6687 DEFUN ("encode-coding-region", Fencode_coding_region
, Sencode_coding_region
,
6688 3, 3, "r\nzCoding system: ",
6689 doc
: /* Encode the current region into the specified coding system.
6690 When called from a program, takes three arguments:
6691 START, END, and CODING-SYSTEM. START and END are buffer positions.
6692 This function sets `last-coding-system-used' to the precise coding system
6693 used (which may be different from CODING-SYSTEM if CODING-SYSTEM is
6694 not fully specified.)
6695 It returns the length of the encoded text. */)
6696 (start
, end
, coding_system
)
6697 Lisp_Object start
, end
, coding_system
;
6699 return code_convert_region1 (start
, end
, coding_system
, 1);
6703 code_convert_string1 (string
, coding_system
, nocopy
, encodep
)
6704 Lisp_Object string
, coding_system
, nocopy
;
6707 struct coding_system coding
;
6709 CHECK_STRING (string
);
6710 CHECK_SYMBOL (coding_system
);
6712 if (NILP (coding_system
))
6713 return (NILP (nocopy
) ? Fcopy_sequence (string
) : string
);
6715 if (setup_coding_system (Fcheck_coding_system (coding_system
), &coding
) < 0)
6716 error ("Invalid coding system: %s", SDATA (SYMBOL_NAME (coding_system
)));
6718 coding
.mode
|= CODING_MODE_LAST_BLOCK
;
6720 ? encode_coding_string (string
, &coding
, !NILP (nocopy
))
6721 : decode_coding_string (string
, &coding
, !NILP (nocopy
)));
6722 Vlast_coding_system_used
= coding
.symbol
;
6727 DEFUN ("decode-coding-string", Fdecode_coding_string
, Sdecode_coding_string
,
6729 doc
: /* Decode STRING which is encoded in CODING-SYSTEM, and return the result.
6730 Optional arg NOCOPY non-nil means it is OK to return STRING itself
6731 if the decoding operation is trivial.
6732 This function sets `last-coding-system-used' to the precise coding system
6733 used (which may be different from CODING-SYSTEM if CODING-SYSTEM is
6734 not fully specified.) */)
6735 (string
, coding_system
, nocopy
)
6736 Lisp_Object string
, coding_system
, nocopy
;
6738 return code_convert_string1 (string
, coding_system
, nocopy
, 0);
6741 DEFUN ("encode-coding-string", Fencode_coding_string
, Sencode_coding_string
,
6743 doc
: /* Encode STRING to CODING-SYSTEM, and return the result.
6744 Optional arg NOCOPY non-nil means it is OK to return STRING itself
6745 if the encoding operation is trivial.
6746 This function sets `last-coding-system-used' to the precise coding system
6747 used (which may be different from CODING-SYSTEM if CODING-SYSTEM is
6748 not fully specified.) */)
6749 (string
, coding_system
, nocopy
)
6750 Lisp_Object string
, coding_system
, nocopy
;
6752 return code_convert_string1 (string
, coding_system
, nocopy
, 1);
6755 /* Encode or decode STRING according to CODING_SYSTEM.
6756 Do not set Vlast_coding_system_used.
6758 This function is called only from macros DECODE_FILE and
6759 ENCODE_FILE, thus we ignore character composition. */
6762 code_convert_string_norecord (string
, coding_system
, encodep
)
6763 Lisp_Object string
, coding_system
;
6766 struct coding_system coding
;
6768 CHECK_STRING (string
);
6769 CHECK_SYMBOL (coding_system
);
6771 if (NILP (coding_system
))
6774 if (setup_coding_system (Fcheck_coding_system (coding_system
), &coding
) < 0)
6775 error ("Invalid coding system: %s", SDATA (SYMBOL_NAME (coding_system
)));
6777 coding
.composing
= COMPOSITION_DISABLED
;
6778 coding
.mode
|= CODING_MODE_LAST_BLOCK
;
6780 ? encode_coding_string (string
, &coding
, 1)
6781 : decode_coding_string (string
, &coding
, 1));
6784 DEFUN ("decode-sjis-char", Fdecode_sjis_char
, Sdecode_sjis_char
, 1, 1, 0,
6785 doc
: /* Decode a Japanese character which has CODE in shift_jis encoding.
6786 Return the corresponding character. */)
6790 unsigned char c1
, c2
, s1
, s2
;
6793 CHECK_NUMBER (code
);
6794 s1
= (XFASTINT (code
)) >> 8, s2
= (XFASTINT (code
)) & 0xFF;
6798 XSETFASTINT (val
, s2
);
6799 else if (s2
>= 0xA0 || s2
<= 0xDF)
6800 XSETFASTINT (val
, MAKE_CHAR (charset_katakana_jisx0201
, s2
, 0));
6802 error ("Invalid Shift JIS code: %x", XFASTINT (code
));
6806 if ((s1
< 0x80 || (s1
> 0x9F && s1
< 0xE0) || s1
> 0xEF)
6807 || (s2
< 0x40 || s2
== 0x7F || s2
> 0xFC))
6808 error ("Invalid Shift JIS code: %x", XFASTINT (code
));
6809 DECODE_SJIS (s1
, s2
, c1
, c2
);
6810 XSETFASTINT (val
, MAKE_CHAR (charset_jisx0208
, c1
, c2
));
6815 DEFUN ("encode-sjis-char", Fencode_sjis_char
, Sencode_sjis_char
, 1, 1, 0,
6816 doc
: /* Encode a Japanese character CHAR to shift_jis encoding.
6817 Return the corresponding code in SJIS. */)
6821 int charset
, c1
, c2
, s1
, s2
;
6825 SPLIT_CHAR (XFASTINT (ch
), charset
, c1
, c2
);
6826 if (charset
== CHARSET_ASCII
)
6830 else if (charset
== charset_jisx0208
6831 && c1
> 0x20 && c1
< 0x7F && c2
> 0x20 && c2
< 0x7F)
6833 ENCODE_SJIS (c1
, c2
, s1
, s2
);
6834 XSETFASTINT (val
, (s1
<< 8) | s2
);
6836 else if (charset
== charset_katakana_jisx0201
6837 && c1
> 0x20 && c2
< 0xE0)
6839 XSETFASTINT (val
, c1
| 0x80);
6842 error ("Can't encode to shift_jis: %d", XFASTINT (ch
));
6846 DEFUN ("decode-big5-char", Fdecode_big5_char
, Sdecode_big5_char
, 1, 1, 0,
6847 doc
: /* Decode a Big5 character which has CODE in BIG5 coding system.
6848 Return the corresponding character. */)
6853 unsigned char b1
, b2
, c1
, c2
;
6856 CHECK_NUMBER (code
);
6857 b1
= (XFASTINT (code
)) >> 8, b2
= (XFASTINT (code
)) & 0xFF;
6861 error ("Invalid BIG5 code: %x", XFASTINT (code
));
6866 if ((b1
< 0xA1 || b1
> 0xFE)
6867 || (b2
< 0x40 || (b2
> 0x7E && b2
< 0xA1) || b2
> 0xFE))
6868 error ("Invalid BIG5 code: %x", XFASTINT (code
));
6869 DECODE_BIG5 (b1
, b2
, charset
, c1
, c2
);
6870 XSETFASTINT (val
, MAKE_CHAR (charset
, c1
, c2
));
6875 DEFUN ("encode-big5-char", Fencode_big5_char
, Sencode_big5_char
, 1, 1, 0,
6876 doc
: /* Encode the Big5 character CHAR to BIG5 coding system.
6877 Return the corresponding character code in Big5. */)
6881 int charset
, c1
, c2
, b1
, b2
;
6885 SPLIT_CHAR (XFASTINT (ch
), charset
, c1
, c2
);
6886 if (charset
== CHARSET_ASCII
)
6890 else if ((charset
== charset_big5_1
6891 && (XFASTINT (ch
) >= 0x250a1 && XFASTINT (ch
) <= 0x271ec))
6892 || (charset
== charset_big5_2
6893 && XFASTINT (ch
) >= 0x290a1 && XFASTINT (ch
) <= 0x2bdb2))
6895 ENCODE_BIG5 (charset
, c1
, c2
, b1
, b2
);
6896 XSETFASTINT (val
, (b1
<< 8) | b2
);
6899 error ("Can't encode to Big5: %d", XFASTINT (ch
));
6903 DEFUN ("set-terminal-coding-system-internal",
6904 Fset_terminal_coding_system_internal
,
6905 Sset_terminal_coding_system_internal
, 1, 1, 0,
6906 doc
: /* Internal use only. */)
6908 Lisp_Object coding_system
;
6910 CHECK_SYMBOL (coding_system
);
6911 setup_coding_system (Fcheck_coding_system (coding_system
), &terminal_coding
);
6912 /* We had better not send unsafe characters to terminal. */
6913 terminal_coding
.flags
|= CODING_FLAG_ISO_SAFE
;
6914 /* Character composition should be disabled. */
6915 terminal_coding
.composing
= COMPOSITION_DISABLED
;
6916 /* Error notification should be suppressed. */
6917 terminal_coding
.suppress_error
= 1;
6918 terminal_coding
.src_multibyte
= 1;
6919 terminal_coding
.dst_multibyte
= 0;
6923 DEFUN ("set-safe-terminal-coding-system-internal",
6924 Fset_safe_terminal_coding_system_internal
,
6925 Sset_safe_terminal_coding_system_internal
, 1, 1, 0,
6926 doc
: /* Internal use only. */)
6928 Lisp_Object coding_system
;
6930 CHECK_SYMBOL (coding_system
);
6931 setup_coding_system (Fcheck_coding_system (coding_system
),
6932 &safe_terminal_coding
);
6933 /* Character composition should be disabled. */
6934 safe_terminal_coding
.composing
= COMPOSITION_DISABLED
;
6935 /* Error notification should be suppressed. */
6936 terminal_coding
.suppress_error
= 1;
6937 safe_terminal_coding
.src_multibyte
= 1;
6938 safe_terminal_coding
.dst_multibyte
= 0;
6942 DEFUN ("terminal-coding-system",
6943 Fterminal_coding_system
, Sterminal_coding_system
, 0, 0, 0,
6944 doc
: /* Return coding system specified for terminal output. */)
6947 return terminal_coding
.symbol
;
6950 DEFUN ("set-keyboard-coding-system-internal",
6951 Fset_keyboard_coding_system_internal
,
6952 Sset_keyboard_coding_system_internal
, 1, 1, 0,
6953 doc
: /* Internal use only. */)
6955 Lisp_Object coding_system
;
6957 CHECK_SYMBOL (coding_system
);
6958 setup_coding_system (Fcheck_coding_system (coding_system
), &keyboard_coding
);
6959 /* Character composition should be disabled. */
6960 keyboard_coding
.composing
= COMPOSITION_DISABLED
;
6964 DEFUN ("keyboard-coding-system",
6965 Fkeyboard_coding_system
, Skeyboard_coding_system
, 0, 0, 0,
6966 doc
: /* Return coding system specified for decoding keyboard input. */)
6969 return keyboard_coding
.symbol
;
6973 DEFUN ("find-operation-coding-system", Ffind_operation_coding_system
,
6974 Sfind_operation_coding_system
, 1, MANY
, 0,
6975 doc
: /* Choose a coding system for an operation based on the target name.
6976 The value names a pair of coding systems: (DECODING-SYSTEM . ENCODING-SYSTEM).
6977 DECODING-SYSTEM is the coding system to use for decoding
6978 \(in case OPERATION does decoding), and ENCODING-SYSTEM is the coding system
6979 for encoding (in case OPERATION does encoding).
6981 The first argument OPERATION specifies an I/O primitive:
6982 For file I/O, `insert-file-contents' or `write-region'.
6983 For process I/O, `call-process', `call-process-region', or `start-process'.
6984 For network I/O, `open-network-stream'.
6986 The remaining arguments should be the same arguments that were passed
6987 to the primitive. Depending on which primitive, one of those arguments
6988 is selected as the TARGET. For example, if OPERATION does file I/O,
6989 whichever argument specifies the file name is TARGET.
6991 TARGET has a meaning which depends on OPERATION:
6992 For file I/O, TARGET is a file name.
6993 For process I/O, TARGET is a process name.
6994 For network I/O, TARGET is a service name or a port number
6996 This function looks up what specified for TARGET in,
6997 `file-coding-system-alist', `process-coding-system-alist',
6998 or `network-coding-system-alist' depending on OPERATION.
6999 They may specify a coding system, a cons of coding systems,
7000 or a function symbol to call.
7001 In the last case, we call the function with one argument,
7002 which is a list of all the arguments given to this function.
7004 usage: (find-operation-coding-system OPERATION ARGUMENTS ...) */)
7009 Lisp_Object operation
, target_idx
, target
, val
;
7010 register Lisp_Object chain
;
7013 error ("Too few arguments");
7014 operation
= args
[0];
7015 if (!SYMBOLP (operation
)
7016 || !INTEGERP (target_idx
= Fget (operation
, Qtarget_idx
)))
7017 error ("Invalid first argument");
7018 if (nargs
< 1 + XINT (target_idx
))
7019 error ("Too few arguments for operation: %s",
7020 SDATA (SYMBOL_NAME (operation
)));
7021 /* For write-region, if the 6th argument (i.e. VISIT, the 5th
7022 argument to write-region) is string, it must be treated as a
7023 target file name. */
7024 if (EQ (operation
, Qwrite_region
)
7026 && STRINGP (args
[5]))
7027 target_idx
= make_number (4);
7028 target
= args
[XINT (target_idx
) + 1];
7029 if (!(STRINGP (target
)
7030 || (EQ (operation
, Qopen_network_stream
) && INTEGERP (target
))))
7031 error ("Invalid argument %d", XINT (target_idx
) + 1);
7033 chain
= ((EQ (operation
, Qinsert_file_contents
)
7034 || EQ (operation
, Qwrite_region
))
7035 ? Vfile_coding_system_alist
7036 : (EQ (operation
, Qopen_network_stream
)
7037 ? Vnetwork_coding_system_alist
7038 : Vprocess_coding_system_alist
));
7042 for (; CONSP (chain
); chain
= XCDR (chain
))
7048 && ((STRINGP (target
)
7049 && STRINGP (XCAR (elt
))
7050 && fast_string_match (XCAR (elt
), target
) >= 0)
7051 || (INTEGERP (target
) && EQ (target
, XCAR (elt
)))))
7054 /* Here, if VAL is both a valid coding system and a valid
7055 function symbol, we return VAL as a coding system. */
7058 if (! SYMBOLP (val
))
7060 if (! NILP (Fcoding_system_p (val
)))
7061 return Fcons (val
, val
);
7062 if (! NILP (Ffboundp (val
)))
7064 val
= call1 (val
, Flist (nargs
, args
));
7067 if (SYMBOLP (val
) && ! NILP (Fcoding_system_p (val
)))
7068 return Fcons (val
, val
);
7076 DEFUN ("update-coding-systems-internal", Fupdate_coding_systems_internal
,
7077 Supdate_coding_systems_internal
, 0, 0, 0,
7078 doc
: /* Update internal database for ISO2022 and CCL based coding systems.
7079 When values of any coding categories are changed, you must
7080 call this function. */)
7085 for (i
= CODING_CATEGORY_IDX_EMACS_MULE
; i
< CODING_CATEGORY_IDX_MAX
; i
++)
7089 val
= SYMBOL_VALUE (XVECTOR (Vcoding_category_table
)->contents
[i
]);
7092 if (! coding_system_table
[i
])
7093 coding_system_table
[i
] = ((struct coding_system
*)
7094 xmalloc (sizeof (struct coding_system
)));
7095 setup_coding_system (val
, coding_system_table
[i
]);
7097 else if (coding_system_table
[i
])
7099 xfree (coding_system_table
[i
]);
7100 coding_system_table
[i
] = NULL
;
7107 DEFUN ("set-coding-priority-internal", Fset_coding_priority_internal
,
7108 Sset_coding_priority_internal
, 0, 0, 0,
7109 doc
: /* Update internal database for the current value of `coding-category-list'.
7110 This function is internal use only. */)
7116 val
= Vcoding_category_list
;
7118 while (CONSP (val
) && i
< CODING_CATEGORY_IDX_MAX
)
7120 if (! SYMBOLP (XCAR (val
)))
7122 idx
= XFASTINT (Fget (XCAR (val
), Qcoding_category_index
));
7123 if (idx
>= CODING_CATEGORY_IDX_MAX
)
7125 coding_priorities
[i
++] = (1 << idx
);
7128 /* If coding-category-list is valid and contains all coding
7129 categories, `i' should be CODING_CATEGORY_IDX_MAX now. If not,
7130 the following code saves Emacs from crashing. */
7131 while (i
< CODING_CATEGORY_IDX_MAX
)
7132 coding_priorities
[i
++] = CODING_CATEGORY_MASK_RAW_TEXT
;
7140 /*** 9. Post-amble ***/
7147 /* Emacs' internal format specific initialize routine. */
7148 for (i
= 0; i
<= 0x20; i
++)
7149 emacs_code_class
[i
] = EMACS_control_code
;
7150 emacs_code_class
[0x0A] = EMACS_linefeed_code
;
7151 emacs_code_class
[0x0D] = EMACS_carriage_return_code
;
7152 for (i
= 0x21 ; i
< 0x7F; i
++)
7153 emacs_code_class
[i
] = EMACS_ascii_code
;
7154 emacs_code_class
[0x7F] = EMACS_control_code
;
7155 for (i
= 0x80; i
< 0xFF; i
++)
7156 emacs_code_class
[i
] = EMACS_invalid_code
;
7157 emacs_code_class
[LEADING_CODE_PRIVATE_11
] = EMACS_leading_code_3
;
7158 emacs_code_class
[LEADING_CODE_PRIVATE_12
] = EMACS_leading_code_3
;
7159 emacs_code_class
[LEADING_CODE_PRIVATE_21
] = EMACS_leading_code_4
;
7160 emacs_code_class
[LEADING_CODE_PRIVATE_22
] = EMACS_leading_code_4
;
7162 /* ISO2022 specific initialize routine. */
7163 for (i
= 0; i
< 0x20; i
++)
7164 iso_code_class
[i
] = ISO_control_0
;
7165 for (i
= 0x21; i
< 0x7F; i
++)
7166 iso_code_class
[i
] = ISO_graphic_plane_0
;
7167 for (i
= 0x80; i
< 0xA0; i
++)
7168 iso_code_class
[i
] = ISO_control_1
;
7169 for (i
= 0xA1; i
< 0xFF; i
++)
7170 iso_code_class
[i
] = ISO_graphic_plane_1
;
7171 iso_code_class
[0x20] = iso_code_class
[0x7F] = ISO_0x20_or_0x7F
;
7172 iso_code_class
[0xA0] = iso_code_class
[0xFF] = ISO_0xA0_or_0xFF
;
7173 iso_code_class
[ISO_CODE_CR
] = ISO_carriage_return
;
7174 iso_code_class
[ISO_CODE_SO
] = ISO_shift_out
;
7175 iso_code_class
[ISO_CODE_SI
] = ISO_shift_in
;
7176 iso_code_class
[ISO_CODE_SS2_7
] = ISO_single_shift_2_7
;
7177 iso_code_class
[ISO_CODE_ESC
] = ISO_escape
;
7178 iso_code_class
[ISO_CODE_SS2
] = ISO_single_shift_2
;
7179 iso_code_class
[ISO_CODE_SS3
] = ISO_single_shift_3
;
7180 iso_code_class
[ISO_CODE_CSI
] = ISO_control_sequence_introducer
;
7182 setup_coding_system (Qnil
, &keyboard_coding
);
7183 setup_coding_system (Qnil
, &terminal_coding
);
7184 setup_coding_system (Qnil
, &safe_terminal_coding
);
7185 setup_coding_system (Qnil
, &default_buffer_file_coding
);
7187 bzero (coding_system_table
, sizeof coding_system_table
);
7189 bzero (ascii_skip_code
, sizeof ascii_skip_code
);
7190 for (i
= 0; i
< 128; i
++)
7191 ascii_skip_code
[i
] = 1;
7193 #if defined (MSDOS) || defined (WINDOWSNT)
7194 system_eol_type
= CODING_EOL_CRLF
;
7196 system_eol_type
= CODING_EOL_LF
;
7199 inhibit_pre_post_conversion
= 0;
7207 Qtarget_idx
= intern ("target-idx");
7208 staticpro (&Qtarget_idx
);
7210 Qcoding_system_history
= intern ("coding-system-history");
7211 staticpro (&Qcoding_system_history
);
7212 Fset (Qcoding_system_history
, Qnil
);
7214 /* Target FILENAME is the first argument. */
7215 Fput (Qinsert_file_contents
, Qtarget_idx
, make_number (0));
7216 /* Target FILENAME is the third argument. */
7217 Fput (Qwrite_region
, Qtarget_idx
, make_number (2));
7219 Qcall_process
= intern ("call-process");
7220 staticpro (&Qcall_process
);
7221 /* Target PROGRAM is the first argument. */
7222 Fput (Qcall_process
, Qtarget_idx
, make_number (0));
7224 Qcall_process_region
= intern ("call-process-region");
7225 staticpro (&Qcall_process_region
);
7226 /* Target PROGRAM is the third argument. */
7227 Fput (Qcall_process_region
, Qtarget_idx
, make_number (2));
7229 Qstart_process
= intern ("start-process");
7230 staticpro (&Qstart_process
);
7231 /* Target PROGRAM is the third argument. */
7232 Fput (Qstart_process
, Qtarget_idx
, make_number (2));
7234 Qopen_network_stream
= intern ("open-network-stream");
7235 staticpro (&Qopen_network_stream
);
7236 /* Target SERVICE is the fourth argument. */
7237 Fput (Qopen_network_stream
, Qtarget_idx
, make_number (3));
7239 Qcoding_system
= intern ("coding-system");
7240 staticpro (&Qcoding_system
);
7242 Qeol_type
= intern ("eol-type");
7243 staticpro (&Qeol_type
);
7245 Qbuffer_file_coding_system
= intern ("buffer-file-coding-system");
7246 staticpro (&Qbuffer_file_coding_system
);
7248 Qpost_read_conversion
= intern ("post-read-conversion");
7249 staticpro (&Qpost_read_conversion
);
7251 Qpre_write_conversion
= intern ("pre-write-conversion");
7252 staticpro (&Qpre_write_conversion
);
7254 Qno_conversion
= intern ("no-conversion");
7255 staticpro (&Qno_conversion
);
7257 Qundecided
= intern ("undecided");
7258 staticpro (&Qundecided
);
7260 Qcoding_system_p
= intern ("coding-system-p");
7261 staticpro (&Qcoding_system_p
);
7263 Qcoding_system_error
= intern ("coding-system-error");
7264 staticpro (&Qcoding_system_error
);
7266 Fput (Qcoding_system_error
, Qerror_conditions
,
7267 Fcons (Qcoding_system_error
, Fcons (Qerror
, Qnil
)));
7268 Fput (Qcoding_system_error
, Qerror_message
,
7269 build_string ("Invalid coding system"));
7271 Qcoding_category
= intern ("coding-category");
7272 staticpro (&Qcoding_category
);
7273 Qcoding_category_index
= intern ("coding-category-index");
7274 staticpro (&Qcoding_category_index
);
7276 Vcoding_category_table
7277 = Fmake_vector (make_number (CODING_CATEGORY_IDX_MAX
), Qnil
);
7278 staticpro (&Vcoding_category_table
);
7281 for (i
= 0; i
< CODING_CATEGORY_IDX_MAX
; i
++)
7283 XVECTOR (Vcoding_category_table
)->contents
[i
]
7284 = intern (coding_category_name
[i
]);
7285 Fput (XVECTOR (Vcoding_category_table
)->contents
[i
],
7286 Qcoding_category_index
, make_number (i
));
7290 Qtranslation_table
= intern ("translation-table");
7291 staticpro (&Qtranslation_table
);
7292 Fput (Qtranslation_table
, Qchar_table_extra_slots
, make_number (1));
7294 Qtranslation_table_id
= intern ("translation-table-id");
7295 staticpro (&Qtranslation_table_id
);
7297 Qtranslation_table_for_decode
= intern ("translation-table-for-decode");
7298 staticpro (&Qtranslation_table_for_decode
);
7300 Qtranslation_table_for_encode
= intern ("translation-table-for-encode");
7301 staticpro (&Qtranslation_table_for_encode
);
7303 Qsafe_chars
= intern ("safe-chars");
7304 staticpro (&Qsafe_chars
);
7306 Qchar_coding_system
= intern ("char-coding-system");
7307 staticpro (&Qchar_coding_system
);
7309 /* Intern this now in case it isn't already done.
7310 Setting this variable twice is harmless.
7311 But don't staticpro it here--that is done in alloc.c. */
7312 Qchar_table_extra_slots
= intern ("char-table-extra-slots");
7313 Fput (Qsafe_chars
, Qchar_table_extra_slots
, make_number (0));
7314 Fput (Qchar_coding_system
, Qchar_table_extra_slots
, make_number (2));
7316 Qvalid_codes
= intern ("valid-codes");
7317 staticpro (&Qvalid_codes
);
7319 Qemacs_mule
= intern ("emacs-mule");
7320 staticpro (&Qemacs_mule
);
7322 Qraw_text
= intern ("raw-text");
7323 staticpro (&Qraw_text
);
7325 defsubr (&Scoding_system_p
);
7326 defsubr (&Sread_coding_system
);
7327 defsubr (&Sread_non_nil_coding_system
);
7328 defsubr (&Scheck_coding_system
);
7329 defsubr (&Sdetect_coding_region
);
7330 defsubr (&Sdetect_coding_string
);
7331 defsubr (&Sfind_coding_systems_region_internal
);
7332 defsubr (&Sunencodable_char_position
);
7333 defsubr (&Sdecode_coding_region
);
7334 defsubr (&Sencode_coding_region
);
7335 defsubr (&Sdecode_coding_string
);
7336 defsubr (&Sencode_coding_string
);
7337 defsubr (&Sdecode_sjis_char
);
7338 defsubr (&Sencode_sjis_char
);
7339 defsubr (&Sdecode_big5_char
);
7340 defsubr (&Sencode_big5_char
);
7341 defsubr (&Sset_terminal_coding_system_internal
);
7342 defsubr (&Sset_safe_terminal_coding_system_internal
);
7343 defsubr (&Sterminal_coding_system
);
7344 defsubr (&Sset_keyboard_coding_system_internal
);
7345 defsubr (&Skeyboard_coding_system
);
7346 defsubr (&Sfind_operation_coding_system
);
7347 defsubr (&Supdate_coding_systems_internal
);
7348 defsubr (&Sset_coding_priority_internal
);
7350 DEFVAR_LISP ("coding-system-list", &Vcoding_system_list
,
7351 doc
: /* List of coding systems.
7353 Do not alter the value of this variable manually. This variable should be
7354 updated by the functions `make-coding-system' and
7355 `define-coding-system-alias'. */);
7356 Vcoding_system_list
= Qnil
;
7358 DEFVAR_LISP ("coding-system-alist", &Vcoding_system_alist
,
7359 doc
: /* Alist of coding system names.
7360 Each element is one element list of coding system name.
7361 This variable is given to `completing-read' as TABLE argument.
7363 Do not alter the value of this variable manually. This variable should be
7364 updated by the functions `make-coding-system' and
7365 `define-coding-system-alias'. */);
7366 Vcoding_system_alist
= Qnil
;
7368 DEFVAR_LISP ("coding-category-list", &Vcoding_category_list
,
7369 doc
: /* List of coding-categories (symbols) ordered by priority.
7371 On detecting a coding system, Emacs tries code detection algorithms
7372 associated with each coding-category one by one in this order. When
7373 one algorithm agrees with a byte sequence of source text, the coding
7374 system bound to the corresponding coding-category is selected. */);
7378 Vcoding_category_list
= Qnil
;
7379 for (i
= CODING_CATEGORY_IDX_MAX
- 1; i
>= 0; i
--)
7380 Vcoding_category_list
7381 = Fcons (XVECTOR (Vcoding_category_table
)->contents
[i
],
7382 Vcoding_category_list
);
7385 DEFVAR_LISP ("coding-system-for-read", &Vcoding_system_for_read
,
7386 doc
: /* Specify the coding system for read operations.
7387 It is useful to bind this variable with `let', but do not set it globally.
7388 If the value is a coding system, it is used for decoding on read operation.
7389 If not, an appropriate element is used from one of the coding system alists:
7390 There are three such tables, `file-coding-system-alist',
7391 `process-coding-system-alist', and `network-coding-system-alist'. */);
7392 Vcoding_system_for_read
= Qnil
;
7394 DEFVAR_LISP ("coding-system-for-write", &Vcoding_system_for_write
,
7395 doc
: /* Specify the coding system for write operations.
7396 Programs bind this variable with `let', but you should not set it globally.
7397 If the value is a coding system, it is used for encoding of output,
7398 when writing it to a file and when sending it to a file or subprocess.
7400 If this does not specify a coding system, an appropriate element
7401 is used from one of the coding system alists:
7402 There are three such tables, `file-coding-system-alist',
7403 `process-coding-system-alist', and `network-coding-system-alist'.
7404 For output to files, if the above procedure does not specify a coding system,
7405 the value of `buffer-file-coding-system' is used. */);
7406 Vcoding_system_for_write
= Qnil
;
7408 DEFVAR_LISP ("last-coding-system-used", &Vlast_coding_system_used
,
7409 doc
: /* Coding system used in the latest file or process I/O. */);
7410 Vlast_coding_system_used
= Qnil
;
7412 DEFVAR_BOOL ("inhibit-eol-conversion", &inhibit_eol_conversion
,
7413 doc
: /* *Non-nil means always inhibit code conversion of end-of-line format.
7414 See info node `Coding Systems' and info node `Text and Binary' concerning
7415 such conversion. */);
7416 inhibit_eol_conversion
= 0;
7418 DEFVAR_BOOL ("inherit-process-coding-system", &inherit_process_coding_system
,
7419 doc
: /* Non-nil means process buffer inherits coding system of process output.
7420 Bind it to t if the process output is to be treated as if it were a file
7421 read from some filesystem. */);
7422 inherit_process_coding_system
= 0;
7424 DEFVAR_LISP ("file-coding-system-alist", &Vfile_coding_system_alist
,
7425 doc
: /* Alist to decide a coding system to use for a file I/O operation.
7426 The format is ((PATTERN . VAL) ...),
7427 where PATTERN is a regular expression matching a file name,
7428 VAL is a coding system, a cons of coding systems, or a function symbol.
7429 If VAL is a coding system, it is used for both decoding and encoding
7431 If VAL is a cons of coding systems, the car part is used for decoding,
7432 and the cdr part is used for encoding.
7433 If VAL is a function symbol, the function must return a coding system
7434 or a cons of coding systems which are used as above. The function gets
7435 the arguments with which `find-operation-coding-system' was called.
7437 See also the function `find-operation-coding-system'
7438 and the variable `auto-coding-alist'. */);
7439 Vfile_coding_system_alist
= Qnil
;
7441 DEFVAR_LISP ("process-coding-system-alist", &Vprocess_coding_system_alist
,
7442 doc
: /* Alist to decide a coding system to use for a process I/O operation.
7443 The format is ((PATTERN . VAL) ...),
7444 where PATTERN is a regular expression matching a program name,
7445 VAL is a coding system, a cons of coding systems, or a function symbol.
7446 If VAL is a coding system, it is used for both decoding what received
7447 from the program and encoding what sent to the program.
7448 If VAL is a cons of coding systems, the car part is used for decoding,
7449 and the cdr part is used for encoding.
7450 If VAL is a function symbol, the function must return a coding system
7451 or a cons of coding systems which are used as above.
7453 See also the function `find-operation-coding-system'. */);
7454 Vprocess_coding_system_alist
= Qnil
;
7456 DEFVAR_LISP ("network-coding-system-alist", &Vnetwork_coding_system_alist
,
7457 doc
: /* Alist to decide a coding system to use for a network I/O operation.
7458 The format is ((PATTERN . VAL) ...),
7459 where PATTERN is a regular expression matching a network service name
7460 or is a port number to connect to,
7461 VAL is a coding system, a cons of coding systems, or a function symbol.
7462 If VAL is a coding system, it is used for both decoding what received
7463 from the network stream and encoding what sent to the network stream.
7464 If VAL is a cons of coding systems, the car part is used for decoding,
7465 and the cdr part is used for encoding.
7466 If VAL is a function symbol, the function must return a coding system
7467 or a cons of coding systems which are used as above.
7469 See also the function `find-operation-coding-system'. */);
7470 Vnetwork_coding_system_alist
= Qnil
;
7472 DEFVAR_LISP ("locale-coding-system", &Vlocale_coding_system
,
7473 doc
: /* Coding system to use with system messages.
7474 Also used for decoding keyboard input on X Window system. */);
7475 Vlocale_coding_system
= Qnil
;
7477 /* The eol mnemonics are reset in startup.el system-dependently. */
7478 DEFVAR_LISP ("eol-mnemonic-unix", &eol_mnemonic_unix
,
7479 doc
: /* *String displayed in mode line for UNIX-like (LF) end-of-line format. */);
7480 eol_mnemonic_unix
= build_string (":");
7482 DEFVAR_LISP ("eol-mnemonic-dos", &eol_mnemonic_dos
,
7483 doc
: /* *String displayed in mode line for DOS-like (CRLF) end-of-line format. */);
7484 eol_mnemonic_dos
= build_string ("\\");
7486 DEFVAR_LISP ("eol-mnemonic-mac", &eol_mnemonic_mac
,
7487 doc
: /* *String displayed in mode line for MAC-like (CR) end-of-line format. */);
7488 eol_mnemonic_mac
= build_string ("/");
7490 DEFVAR_LISP ("eol-mnemonic-undecided", &eol_mnemonic_undecided
,
7491 doc
: /* *String displayed in mode line when end-of-line format is not yet determined. */);
7492 eol_mnemonic_undecided
= build_string (":");
7494 DEFVAR_LISP ("enable-character-translation", &Venable_character_translation
,
7495 doc
: /* *Non-nil enables character translation while encoding and decoding. */);
7496 Venable_character_translation
= Qt
;
7498 DEFVAR_LISP ("standard-translation-table-for-decode",
7499 &Vstandard_translation_table_for_decode
,
7500 doc
: /* Table for translating characters while decoding. */);
7501 Vstandard_translation_table_for_decode
= Qnil
;
7503 DEFVAR_LISP ("standard-translation-table-for-encode",
7504 &Vstandard_translation_table_for_encode
,
7505 doc
: /* Table for translating characters while encoding. */);
7506 Vstandard_translation_table_for_encode
= Qnil
;
7508 DEFVAR_LISP ("charset-revision-table", &Vcharset_revision_alist
,
7509 doc
: /* Alist of charsets vs revision numbers.
7510 While encoding, if a charset (car part of an element) is found,
7511 designate it with the escape sequence identifying revision (cdr part of the element). */);
7512 Vcharset_revision_alist
= Qnil
;
7514 DEFVAR_LISP ("default-process-coding-system",
7515 &Vdefault_process_coding_system
,
7516 doc
: /* Cons of coding systems used for process I/O by default.
7517 The car part is used for decoding a process output,
7518 the cdr part is used for encoding a text to be sent to a process. */);
7519 Vdefault_process_coding_system
= Qnil
;
7521 DEFVAR_LISP ("latin-extra-code-table", &Vlatin_extra_code_table
,
7522 doc
: /* Table of extra Latin codes in the range 128..159 (inclusive).
7523 This is a vector of length 256.
7524 If Nth element is non-nil, the existence of code N in a file
7525 \(or output of subprocess) doesn't prevent it to be detected as
7526 a coding system of ISO 2022 variant which has a flag
7527 `accept-latin-extra-code' t (e.g. iso-latin-1) on reading a file
7528 or reading output of a subprocess.
7529 Only 128th through 159th elements has a meaning. */);
7530 Vlatin_extra_code_table
= Fmake_vector (make_number (256), Qnil
);
7532 DEFVAR_LISP ("select-safe-coding-system-function",
7533 &Vselect_safe_coding_system_function
,
7534 doc
: /* Function to call to select safe coding system for encoding a text.
7536 If set, this function is called to force a user to select a proper
7537 coding system which can encode the text in the case that a default
7538 coding system used in each operation can't encode the text.
7540 The default value is `select-safe-coding-system' (which see). */);
7541 Vselect_safe_coding_system_function
= Qnil
;
7543 DEFVAR_LISP ("char-coding-system-table", &Vchar_coding_system_table
,
7544 doc
: /* Char-table containing safe coding systems of each characters.
7545 Each element doesn't include such generic coding systems that can
7546 encode any characters. They are in the first extra slot. */);
7547 Vchar_coding_system_table
= Fmake_char_table (Qchar_coding_system
, Qnil
);
7549 DEFVAR_BOOL ("inhibit-iso-escape-detection",
7550 &inhibit_iso_escape_detection
,
7551 doc
: /* If non-nil, Emacs ignores ISO2022's escape sequence on code detection.
7553 By default, on reading a file, Emacs tries to detect how the text is
7554 encoded. This code detection is sensitive to escape sequences. If
7555 the sequence is valid as ISO2022, the code is determined as one of
7556 the ISO2022 encodings, and the file is decoded by the corresponding
7557 coding system (e.g. `iso-2022-7bit').
7559 However, there may be a case that you want to read escape sequences in
7560 a file as is. In such a case, you can set this variable to non-nil.
7561 Then, as the code detection ignores any escape sequences, no file is
7562 detected as encoded in some ISO2022 encoding. The result is that all
7563 escape sequences become visible in a buffer.
7565 The default value is nil, and it is strongly recommended not to change
7566 it. That is because many Emacs Lisp source files that contain
7567 non-ASCII characters are encoded by the coding system `iso-2022-7bit'
7568 in Emacs's distribution, and they won't be decoded correctly on
7569 reading if you suppress escape sequence detection.
7571 The other way to read escape sequences in a file without decoding is
7572 to explicitly specify some coding system that doesn't use ISO2022's
7573 escape sequence (e.g `latin-1') on reading by \\[universal-coding-system-argument]. */);
7574 inhibit_iso_escape_detection
= 0;
7578 emacs_strerror (error_number
)
7583 synchronize_system_messages_locale ();
7584 str
= strerror (error_number
);
7586 if (! NILP (Vlocale_coding_system
))
7588 Lisp_Object dec
= code_convert_string_norecord (build_string (str
),
7589 Vlocale_coding_system
,
7591 str
= (char *) SDATA (dec
);