[bpt/emacs.git] / admin / notes / unicode

                                            -*-mode: text; coding: utf-8;-*-

Copyright (C) 2002-2013 Free Software Foundation, Inc.
See the end of the file for license conditions.

Problems, fixmes and other unicode-related issues
-------------------------------------------------------------

Notes by fx to record various things of variable importance.  handa
needs to check them -- don't take too seriously, especially with
regard to completeness.

 * SINGLE_BYTE_CHAR_P returns true for Latin-1 characters, which has
   undesirable effects.  E.g.:
   (multibyte-string-p (let ((s "x")) (aset s 0 ?£) s)) => nil
   (multibyte-string-p (concat [?£])) => nil
   (text-char-description ?£) => "M-#"

	These examples are all fixed by the change of 2002-10-14, but
	there still exist questionable SINGLE_BYTE_CHAR_P in the
	code (keymap.c and print.c).

 * Rationalize character syntax and its relationship to the Unicode
   database.  (Applies mainly to symbol an punctuation syntax.)

 * Fontset handling and customization needs work.  We want to relate
   fonts to scripts, probably based on the Unicode blocks.  The
   presence of small-repertoire 10646-encoded fonts in XFree 4 is a
   pain, not currently worked round.

	With the change on 2002-07-26, multiple fonts can be
	specified in a fontset for a specific range of characters.
	Each range can also be specified by script.  Before using
	ISO10646 fonts, Emacs checks their repertories to avoid such
	fonts that don't have a glyph for a specific character.

	fx has worked on fontset customization, but was stymied by
	basic problems with the way the default face is dealt with
	(and something else, I think).  This needs revisiting.

 * Work is also needed on charset and coding system priorities.

 * The relevant bits of latin1-disp.el need porting (and probably
   re-naming/updating).  See also cyril-util.el.

 * Quail files need more work now the encoding is largely irrelevant.

 * What to do with the old coding categories stuff?

 * The preferred-coding-system property of charsets should probably be
   junked unless it can be made more useful now.

 * find-multibyte-characters needs looking at.

 * Implement Korean cp949/UHC, BIG5-HKSCS and any other important missing
   charsets.

 * Lazy-load tables for unify-charset somehow?

	Actually, Emacs clears out all charset maps and unify-map just
	before dumping, and they are loaded again on demand by the
	dumped emacs.  But, those maps (char tables) generated while
	temacs is running can't be removed from the dumped emacs.

 * iso-2022 charsets get unified on i/o.

	With the change on 2003-01-06, decoding routines put `charset'
	property to decoded text, and iso-2022 encoder pay attention
	to it.  Thus, for instance, reading and writing by
	iso-2022-7bit preserve the original designation sequences.
	The property name `preferred-charset' may be better?

	We may have to utilize this property to decide a font.

 * Revisit locale processing: look at treating the language and
   charset parts separately.  (Language should affect things like
   spelling and calendar, but that's not a Unicode issue.)

 * Handle Unicode combining characters usefully, e.g. diacritics, and
   handle more scripts specifically (à la Devanagari).  There are
   issues with canonicalization.

 * We need tabular input methods, e.g. for maths symbols.  (Not
   specific to Unicode.)

 * Need multibyte text in menus, e.g. for the above.  (Not specific to
   Unicode -- see Emacs etc/TODO, but now mostly works with gtk.)

 * There's currently no support for Unicode normalization.

 * Populate char-width-table correctly for Unicode characters and
   worry about what happens when double-width charsets covering
   non-CJK characters are unified.

 * There are type errors lurking, e.g. in
   Fcheck_coding_systems_region.  Define ENABLE_CHECKING to find them.

 * Old auto-save files, and similar files, such as Gnus drafts,
   containing non-ASCII characters probably won't be re-read correctly.


Source file encoding
--------------------

Most Emacs source files are encoded in UTF-8 (or in ASCII, which is a
subset), but there are a few exceptions, listed below.  Perhaps
someday many of these files will be converted to UTF-8, for
convenience when using tools like 'grep -r', but this might need
nontrivial changes to the build process.

 * chinese-big5

     These are verbatim copies of files taken from external sources.
     They haven't been converted to UTF-8.

	leim/CXTERM-DIC/4Corner.tit
	leim/CXTERM-DIC/ARRAY30.tit
	leim/CXTERM-DIC/ECDICT.tit
	leim/CXTERM-DIC/ETZY.tit
	leim/CXTERM-DIC/PY-b5.tit
	leim/CXTERM-DIC/Punct-b5.tit
	leim/CXTERM-DIC/QJ-b5.tit
	leim/CXTERM-DIC/ZOZY.tit
	leim/MISC-DIC/CTLau-b5.html
	leim/MISC-DIC/cangjie-table.b5

 * chinese-iso-8bit

     These are verbatim copies of files taken from external sources.
     They haven't been converted to UTF-8.

	leim/CXTERM-DIC/CCDOSPY.tit
	leim/CXTERM-DIC/Punct.tit
	leim/CXTERM-DIC/QJ.tit
	leim/CXTERM-DIC/SW.tit
	leim/CXTERM-DIC/TONEPY.tit
	leim/MISC-DIC/pinyin.map
	leim/MISC-DIC/CTLau.html
	leim/MISC-DIC/ziranma.cin

 * cp850

     This file contains non-ASCII characters in unibyte strings.  When
     editing a keyboard layout it's more convenient to see 'é' than
     '\202', and the MS-DOS compiler requires the single byte if a
     backslash escape is not being used.

	src/msdos.c

 * iso-2022-cn-ext

     This file is externally generated from leim/MISC-DIC/cangjie-table.b5
     by Big5->CNS converter.  It hasn't been converted to UTF-8.

	leim/MISC-DIC/cangjie-table.cns

 * iso-latin-2

     These files are processed by csplain, a program that requires
     Latin-2 input.  In 2012 the csplain maintainers started
     recommending UTF-8, but these files haven't been converted yet.

	etc/refcards/cs-dired-ref.tex
	etc/refcards/cs-refcard.tex
	etc/refcards/cs-survival.tex
	etc/refcards/sk-dired-ref.tex
	etc/refcards/sk-refcard.tex
	etc/refcards/sk-survival.tex

 * japanese-iso-8bit

     SKK-JISYO.L is a verbatim copy of a file taken from an external source.
     ja-dic.el is generated automatically by skkdic-convert; this process
     hasn't been converted to use UTF-8.

	leim/SKK-DIC/SKK-JISYO.L
	leim/ja-dic/ja-dic.el

 * japanese-shift-jis

     This is a verbatim copy of a file taken from an external source.
     It hasn't been converted to UTF-8.

	admin/charsets/mapfiles/cns2ucsdkw.txt

 * no-conversion

     This file purposely contains arbitrary bytes interspersed within text,
     to test whether the Emacs distribution is corrupted.

	lib-src/testfile

 * iso-2022-7bit

     These files contain characters that cannot be encoded in UTF-8.

	leim/quail/tibetan.el
	leim/quail/ethiopic.el
	lisp/international/titdic-cnv.el
	lisp/language/tibetan.el
	lisp/language/tibet-util.el
	lisp/language/ind-util.el

\f
This file is part of GNU Emacs.

GNU Emacs is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

GNU Emacs is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.
Commit	Line	Data
c38e0c97	1	--mode: text; coding: utf-8;--
e88a2ed3	2
ab422c4d	3	Copyright (C) 2002-2013 Free Software Foundation, Inc.
e88a2ed3 GM	4	See the end of the file for license conditions.
	5
	6	Problems, fixmes and other unicode-related issues
	7	-------------------------------------------------------------
	8
	9	Notes by fx to record various things of variable importance. handa
	10	needs to check them -- don't take too seriously, especially with
	11	regard to completeness.
	12
	13	* SINGLE_BYTE_CHAR_P returns true for Latin-1 characters, which has
	14	undesirable effects. E.g.:
c38e0c97 PE	15	(multibyte-string-p (let ((s "x")) (aset s 0 ?£) s)) => nil
	16	(multibyte-string-p (concat [?£])) => nil
	17	(text-char-description ?£) => "M-#"
e88a2ed3 GM	18
	19	These examples are all fixed by the change of 2002-10-14, but
	20	there still exist questionable SINGLE_BYTE_CHAR_P in the
	21	code (keymap.c and print.c).
	22
	23	* Rationalize character syntax and its relationship to the Unicode
	24	database. (Applies mainly to symbol an punctuation syntax.)
	25
	26	* Fontset handling and customization needs work. We want to relate
	27	fonts to scripts, probably based on the Unicode blocks. The
	28	presence of small-repertoire 10646-encoded fonts in XFree 4 is a
	29	pain, not currently worked round.
	30
	31	With the change on 2002-07-26, multiple fonts can be
	32	specified in a fontset for a specific range of characters.
	33	Each range can also be specified by script. Before using
	34	ISO10646 fonts, Emacs checks their repertories to avoid such
	35	fonts that don't have a glyph for a specific character.
	36
	37	fx has worked on fontset customization, but was stymied by
	38	basic problems with the way the default face is dealt with
	39	(and something else, I think). This needs revisiting.
	40
	41	* Work is also needed on charset and coding system priorities.
	42
	43	* The relevant bits of latin1-disp.el need porting (and probably
	44	re-naming/updating). See also cyril-util.el.
	45
	46	* Quail files need more work now the encoding is largely irrelevant.
	47
	48	* What to do with the old coding categories stuff?
	49
	50	* The preferred-coding-system property of charsets should probably be
	51	junked unless it can be made more useful now.
	52
	53	* find-multibyte-characters needs looking at.
	54
	55	* Implement Korean cp949/UHC, BIG5-HKSCS and any other important missing
	56	charsets.
	57
	58	* Lazy-load tables for unify-charset somehow?
	59
	60	Actually, Emacs clears out all charset maps and unify-map just
	61	before dumping, and they are loaded again on demand by the
	62	dumped emacs. But, those maps (char tables) generated while
	63	temacs is running can't be removed from the dumped emacs.
	64
e88a2ed3 GM	65	* iso-2022 charsets get unified on i/o.
	66
	67	With the change on 2003-01-06, decoding routines put `charset'
	68	property to decoded text, and iso-2022 encoder pay attention
	69	to it. Thus, for instance, reading and writing by
	70	iso-2022-7bit preserve the original designation sequences.
	71	The property name `preferred-charset' may be better?
	72
	73	We may have to utilize this property to decide a font.
	74
	75	* Revisit locale processing: look at treating the language and
	76	charset parts separately. (Language should affect things like
	77	spelling and calendar, but that's not a Unicode issue.)
	78
	79	* Handle Unicode combining characters usefully, e.g. diacritics, and
c38e0c97	80	handle more scripts specifically (à la Devanagari). There are
e88a2ed3 GM	81	issues with canonicalization.
e88a2ed3 GM	82
e88a2ed3 GM	83	* We need tabular input methods, e.g. for maths symbols. (Not
	84	specific to Unicode.)
	85
	86	* Need multibyte text in menus, e.g. for the above. (Not specific to
	87	Unicode -- see Emacs etc/TODO, but now mostly works with gtk.)
	88
	89	* There's currently no support for Unicode normalization.
	90
	91	* Populate char-width-table correctly for Unicode characters and
	92	worry about what happens when double-width charsets covering
	93	non-CJK characters are unified.
	94
e88a2ed3 GM	95	* There are type errors lurking, e.g. in
	96	Fcheck_coding_systems_region. Define ENABLE_CHECKING to find them.
	97
e88a2ed3 GM	98	* Old auto-save files, and similar files, such as Gnus drafts,
	99	containing non-ASCII characters probably won't be re-read correctly.
	100
d37e4893 PE	101
	102	Source file encoding
	103	--------------------
	104
	105	Most Emacs source files are encoded in UTF-8 (or in ASCII, which is a
	106	subset), but there are a few exceptions, listed below. Perhaps
2b0fae5e	107	someday many of these files will be converted to UTF-8, for
1b610f51 PE	108	convenience when using tools like 'grep -r', but this might need
1b610f51 PE	109	nontrivial changes to the build process.
d37e4893 PE	110
	111	* chinese-big5
	112
1b610f51 PE	113	These are verbatim copies of files taken from external sources.
	114	They haven't been converted to UTF-8.
	115
d37e4893 PE	116	leim/CXTERM-DIC/4Corner.tit
	117	leim/CXTERM-DIC/ARRAY30.tit
	118	leim/CXTERM-DIC/ECDICT.tit
	119	leim/CXTERM-DIC/ETZY.tit
	120	leim/CXTERM-DIC/PY-b5.tit
	121	leim/CXTERM-DIC/Punct-b5.tit
	122	leim/CXTERM-DIC/QJ-b5.tit
	123	leim/CXTERM-DIC/ZOZY.tit
	124	leim/MISC-DIC/CTLau-b5.html
	125	leim/MISC-DIC/cangjie-table.b5
	126
	127	* chinese-iso-8bit
	128
1b610f51 PE	129	These are verbatim copies of files taken from external sources.
	130	They haven't been converted to UTF-8.
	131
d37e4893 PE	132	leim/CXTERM-DIC/CCDOSPY.tit
	133	leim/CXTERM-DIC/Punct.tit
	134	leim/CXTERM-DIC/QJ.tit
	135	leim/CXTERM-DIC/SW.tit
	136	leim/CXTERM-DIC/TONEPY.tit
	137	leim/MISC-DIC/pinyin.map
	138	leim/MISC-DIC/CTLau.html
	139	leim/MISC-DIC/ziranma.cin
	140
1b610f51 PE	141	* cp850
	142
	143	This file contains non-ASCII characters in unibyte strings. When
	144	editing a keyboard layout it's more convenient to see 'é' than
	145	'\202', and the MS-DOS compiler requires the single byte if a
	146	backslash escape is not being used.
	147
	148	src/msdos.c
	149
	150	* iso-2022-cn-ext
	151
	152	This file is externally generated from leim/MISC-DIC/cangjie-table.b5
	153	by Big5->CNS converter. It hasn't been converted to UTF-8.
	154
	155	leim/MISC-DIC/cangjie-table.cns
	156
d37e4893 PE	157	* iso-latin-2
d37e4893 PE	158
1b610f51 PE	159	These files are processed by csplain, a program that requires
	160	Latin-2 input. In 2012 the csplain maintainers started
	161	recommending UTF-8, but these files haven't been converted yet.
	162
	163	etc/refcards/cs-dired-ref.tex
d37e4893	164	etc/refcards/cs-refcard.tex
d37e4893	165	etc/refcards/cs-survival.tex
d37e4893 PE	166	etc/refcards/sk-dired-ref.tex
d37e4893 PE	167	etc/refcards/sk-refcard.tex
1b610f51	168	etc/refcards/sk-survival.tex
d37e4893 PE	169
	170	* japanese-iso-8bit
	171
1b610f51 PE	172	SKK-JISYO.L is a verbatim copy of a file taken from an external source.
	173	ja-dic.el is generated automatically by skkdic-convert; this process
	174	hasn't been converted to use UTF-8.
	175
d37e4893 PE	176	leim/SKK-DIC/SKK-JISYO.L
	177	leim/ja-dic/ja-dic.el
	178
	179	* japanese-shift-jis
	180
1b610f51 PE	181	This is a verbatim copy of a file taken from an external source.
	182	It hasn't been converted to UTF-8.
	183
d37e4893 PE	184	admin/charsets/mapfiles/cns2ucsdkw.txt
	185
	186	* no-conversion
	187
1b610f51 PE	188	This file purposely contains arbitrary bytes interspersed within text,
	189	to test whether the Emacs distribution is corrupted.
	190
d37e4893 PE	191	lib-src/testfile
d37e4893 PE	192
1b610f51 PE	193	* iso-2022-7bit
	194
	195	These files contain characters that cannot be encoded in UTF-8.
	196
	197	leim/quail/tibetan.el
	198	leim/quail/ethiopic.el
	199	lisp/international/titdic-cnv.el
	200	lisp/language/tibetan.el
	201	lisp/language/tibet-util.el
	202	lisp/language/ind-util.el
	203
e88a2ed3 GM	204	\f
	205	This file is part of GNU Emacs.
	206
9ad5de0c	207	GNU Emacs is free software: you can redistribute it and/or modify
e88a2ed3	208	it under the terms of the GNU General Public License as published by
9ad5de0c GM	209	the Free Software Foundation, either version 3 of the License, or
9ad5de0c GM	210	(at your option) any later version.
e88a2ed3 GM	211
	212	GNU Emacs is distributed in the hope that it will be useful,
	213	but WITHOUT ANY WARRANTY; without even the implied warranty of
	214	MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
	215	GNU General Public License for more details.
	216
	217	You should have received a copy of the GNU General Public License
9ad5de0c	218	along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>.