README.unicode

   1                                             -*-mode: text; coding: latin-1;-*-
   2
   3 Problems, fixmes and other issues in the emacs-unicode branch
   4 -------------------------------------------------------------
   5
   6 Notes by fx to record various things of variable importance.  handa
   7 needs to check them -- don't take too seriously, especially with
   8 regard to completeness.
   9
  10 _Do take seriously that you don't want this branch unless you're
  11 actually working on it; you risk your data by actually using it._  If
  12 you just want to edit Unicode and/or unify iso-8859 et al, see the
  13 existing support and the extra stuff at
  14 <URL:ftp://dlpx1.dl.ac.uk/fx/emacs/Mule>, mostly now in the CVS trunk.
  15 (Editing support is mostly orthogonal to the internal representation.)
  16
  17  * SINGLE_BYTE_CHAR_P returns true for Latin-1 characters, which has
  18    undesirable effects.  E.g.:
  19    (multibyte-string-p (let ((s "x")) (aset s 0 ?£) s)) => nil
  20    (multibyte-string-p (concat [?£])) => nil
  21    (text-char-description ?£) => "M-#"
  22
  23         These examples are all fixed by the change of 2002-10-14, but
  24         there still exist questionalble SINGLE_BYTE_CHAR_P in the
  25         code (keymap.c and print.c).
  26
  27  * Rationalize character syntax and its relationship to the Unicode
  28    database.  (Applies mainly to symbol an punctuation syntax.)
  29
  30  * Fontset handling and customization needs work.  We want to relate
  31    fonts to scripts, probably based on the Unicode blocks.  The
  32    presence of small-repertoire 10646-encoded fonts in XFree 4 is a
  33    pain, not currently worked round.
  34
  35         With the change on 2002-07-26, multiple fonts can be
  36         specified in a fontset for a specific range of characters.
  37         Each range can also be specified by script.  Before using
  38         ISO10646 fonts, Emacs checks their repertories to avoid such
  39         fonts that don't have a glyph for a specific character.
  40
  41         fx has worked on fontset customization, but was stymied by
  42         basic problems with the way the default face is dealt with
  43         (and something else, I think).  This needs revisiting.
  44
  45  * Work is also needed on charset and coding system priorities.
  46
  47  * The relevant bits of latin1-disp.el need porting (and probably
  48    re-naming/updating).  See also cyril-util.el.
  49
  50  * Quail files need more work now the encoding is largely irrelevant.
  51
  52  * What to do with the old coding categories stuff?
  53
  54  * The preferred-coding-system property of charsets should probably be
  55    junked unless it can be made more useful now.
  56
  57  * find-multibyte-characters needs looking at.
  58
  59  * Implement Korean cp949/UHC, BIG5-HKSCS and any other important missing
  60    charsets.
  61
  62  * Lazy-load tables for unify-charset somehow?
  63
  64         Actually, Emacs clear out all charset maps and unify-map just
  65         before dumping, and their are loaded again on demand the
  66         dumped emacs.  But, those maps (char tables) generated while
  67         temacs is running can't be get rid of from the dumped emacs.
  68
  69  * Translation tables for {en,de}code currently aren't supported.
  70
  71         This should be fixed by the changes of 2002-10-14.
  72
  73  * Defining CCL coding systems currently doesn't work.
  74
  75         This should be fixed by the changes of 2003-01-30.
  76
  77  * iso-2022 charsets get unified on i/o.
  78
  79         With the change on 2003-01-06, decoding routines put `charset'
  80         property to decoded text, and iso-2022 encoder pay attention
  81         to it.  Thus, for instance, reading and writing by
  82         iso-2022-7bit preserve the original designation sequences.
  83         The property name `preferred-charset' may be better?
  84
  85         We may have to utilize this property to decide a font.
  86
  87  * Revisit locale processing: look at treating the language and
  88    charset parts separately.  (Language should affect things like
  89    speling and calendar, but that's not a Unicode issue.)
  90
  91  * Handle Unicode combining characters usefully, e.g. diacritics, and
  92    handle more scripts specifically (à la Devanagari).  There are
  93    issues with canonicalization.
  94
  95  * Bidi is a separate issue with no support currently.
  96
  97  * We need tabular input methods, e.g. for maths symbols.  (Not
  98    specific to Unicode.)
  99
 100  * Need multibyte text in menus, e.g. for the above.  (Not specific to
 101    Unicode -- see Emacs etc/TODO, but now mostly works with gtk.)
 102
 103  * There's currently no support for Unicode normalization.
 104
 105  * Populate char-width-table correctly for Unicode chanaracters and
 106    worry about what happens when double-width charsets covering
 107    non-CJK characters are unified.
 108
 109  * Emacs 20/21 .elc files are currently not loadable.  It may or may
 110    not be possible to do this properly.
 111
 112         With the change on 2002-07-24, elc files generated by Emacs
 113         20.3 and later are correctly loaded (including those
 114         containing multibyte characters and compressed).  But, elc
 115         files generated by 20.2 and the primer are still not loadable.
 116         Is it really worth working on it?
 117
 118  * Rmail won't work with non-ASCII text.  Encoding issues for Babyl
 119    files need sorting out, but rms says Babyl will go before this is
 120    released.
 121
 122  * Gnus still needs some attention, and we need to get changes
 123    accepted by Gnus maintainers...
 124
 125  * There are type errors lurking, e.g. in
 126    Fcheck_coding_systems_region.  Define ENABLE_CHECKING to find them.
 127
 128  * You can grep the code for lots of fixmes.
 129
 130  * Old auto-save files, and similar files, such as Gnus drafts,
 131    containing non-ASCII characters probably won't be re-read correctly.
 132
 133
 134
 135 New font handling mechanism with font backend method
 136 ----------------------------------------------------
 137
 138 This branch now contains new codes for handling fonts by multiple font
 139 backends.  The old font handling codes still exist completely parallel
 140 to the new codes, and the new codes are used only when you configure
 141 Emacs with the argument "--enable-font-backend" and run Emacs with the
 142 same argument.
 143
 144 Which font backends to use can be specified by X resource
 145 "FontBackend".  For instance, if you want to use Xft fonts only,
 146
 147 Emacs.FontBackend: xft
 148
 149 will work.  If this resource is not set, Emacs tries to use all font
 150 backends available on your graphic device.
 151
 152 The configure script, if invoked with "--enable-font-backend", checks
 153 existing of libraries freetype and fontconfig.  If they are both
 154 available, macro "USE_FONT_BACKEND" is defined in src/config.h.  In
 155 that case, the existing of Xft library is checked too.
 156
 157 The new files are:
 158         font.h -- header providing font-backend related structures
 159                 (most important ones are "struct font" and "struct
 160                 font_driver"), macros, and etc.
 161         font.c -- main font handling code.
 162         xfont.c -- font-driver on X for X core fonts.
 163         ftfont.c -- generic font-driver for FreeType fonts providing
 164                 device-independent methods of struct font_driver.
 165         xftfont.c -- font-driver on X using Xft for FreeType fonts
 166                 utilizing methods provided by ftfont.c.
 167         ftxfont.c -- font-driver on X directly using FreeType fonts
 168                 utilizing methods provided by ftfont.c.
 169
 170 So we already have codes for X.  For the other systems (w32 and mac),
 171 it seems that we need these files:
 172         w32font.c -- font driver on w32 using Windows native fonts,
 173                 corresponding to xfont.c
 174         atmfont.c -- font-driver on mac using ATM fonts, corresponding
 175                 to xfont.c
 176 As BDF fonts are currently used on w32, we may also implement these:
 177         bdffont.c -- generic font-driver for BDF fonts, corresponding to
 178                 ftfont.c
 179         bdfw32font.c -- font-driver on w32 using BDF fonts,
 180                 corresponding to ftxfont.c
 181 But, as FreeType already supports BDF fonts, if FreeType and
 182 Fontconfig are also available on w32, what we need may be:
 183         ftw32font.c -- font-driver on w32 directly using FreeType fonts
 184                 utilizing methods provided by ftfont.c.
 185
 186 And, for those to work, w32term.c (macterm.c) and w32fns.c (macfns.c)
 187 must be changed by the similar way as xterm.c and xfns.c (the parts
 188 "#ifdef USE_FONT_BACKEND" ... "#endif" should be checked).
 189
 190 It may be interesting if Emacs supports a frame buffer directly and
 191 have these font driver.
 192         ftfbfont.c -- font-driver on FB for FreeType fonts.
 193         bdffbfont.c -- font-driver on FB for BDF fonts.
 194
 195 Note: The fontset related codes are not yet matuared to work well with
 196 the font backend method.  So, for instance, even if you start Emacs
 197 as something like this:
 198   % emacs --enable-font-backend -fn tahoma
 199 Non-ASCII Latin characters will not be displayed by the font "tahoma".
 200 In such a case, please try this:
 201
 202 (set-fontset-font "fontset-default" 'latin '("tahoma" . "unicode-bmp"))