| 1 | |
| 2 | |
| 3 | README: library "Sexplib" |
| 4 | ************************* |
| 5 | Copyright (C) 2012 Jane Street Holding, LLC (1) |
| 6 | ===================================================== |
| 7 | Author: Markus Mottl |
| 8 | ====================== |
| 9 | New York, 2012-03-20 |
| 10 | ==================== |
| 11 | |
| 12 | |
| 13 | 1 Directory contents |
| 14 | *=*=*=*=*=*=*=*=*=*=* |
| 15 | |
| 16 | |
| 17 | |
| 18 | ------------------------------------------------------------------------ |
| 19 | | CHANGES | History of code changes | |
| 20 | ------------------------------------------------------------------------ |
| 21 | | COPYRIGHT | Notes on copyright | |
| 22 | ------------------------------------------------------------------------ |
| 23 | | INSTALL | Short notes on compiling and | |
| 24 | | | installing the library | |
| 25 | ------------------------------------------------------------------------ |
| 26 | | LICENSE | "GNU LESSER GENERAL PUBLIC LICENSE" | |
| 27 | ------------------------------------------------------------------------ |
| 28 | | LICENSE.Tywith | License of Tywith, from which Sexplib is derived| |
| 29 | ------------------------------------------------------------------------ |
| 30 | | Makefile | Top Makefile | |
| 31 | ------------------------------------------------------------------------ |
| 32 | | OCamlMakefile | Generic Makefile for OCaml-projects | |
| 33 | ------------------------------------------------------------------------ |
| 34 | | OMakefile | Ignore this file | |
| 35 | ------------------------------------------------------------------------ |
| 36 | | README.txt | This file | |
| 37 | ------------------------------------------------------------------------ |
| 38 | | lib/ | OCaml-library for S-expression conversions | |
| 39 | ------------------------------------------------------------------------ |
| 40 | | lib_test/ | Test applications for the Sexplib-library | |
| 41 | ------------------------------------------------------------------------ |
| 42 | |
| 43 | |
| 44 | |
| 45 | |
| 46 | 2 What is "Sexplib"? |
| 47 | *=*=*=*=*=*=*=*=*=*=* |
| 48 | |
| 49 | |
| 50 | This library contains functionality for parsing and pretty-printing |
| 51 | S-expressions. In addition to that it contains a preprocessing module for |
| 52 | Camlp4, which can be used to automatically generate code from type definitions |
| 53 | for efficiently converting OCaml-values to S-expressions and vice versa. In |
| 54 | combination with the parsing and pretty-printing functionality this frees |
| 55 | users from having to write their own I/O-routines for datastructures they |
| 56 | define. Possible errors during automatic conversions from S-expressions to |
| 57 | OCaml-values are reported in human-readable ways with exact location |
| 58 | information. Another module in this library allows you to extract and replace |
| 59 | sub-expressions in S-expressions. |
| 60 | |
| 61 | |
| 62 | 3 How can you use it? |
| 63 | *=*=*=*=*=*=*=*=*=*=*= |
| 64 | |
| 65 | |
| 66 | Make sure you have installed the 'type_conv' package on your system, too. It |
| 67 | should be obtainable at the same site as 'sexplib'. |
| 68 | |
| 69 | The API (.mli-files) in the 'sexplib' library directory is fully documented. |
| 70 | Module 'Sexp' contains all I/O-functions for S-expressions, module 'Conv' |
| 71 | helper functions for converting OCaml-values of standard types to |
| 72 | S-expressions. Module 'Path' supports sub-expression extraction and |
| 73 | substitution. |
| 74 | |
| 75 | Module 'pa_sexp_conv.ml' contains the extensions for the |
| 76 | Camlp4-preprocessor. It adds the new construct 'with sexp' (and 'with sexp_of' |
| 77 | and 'with of_sexp', which are implied by the first). When using this construct |
| 78 | right after a type definition, function definitions will be generated |
| 79 | automatically, which perform S-expression conversions. |
| 80 | |
| 81 | E.g. given the following type definition: |
| 82 | << type t = A | B |
| 83 | with sexp |
| 84 | >> |
| 85 | |
| 86 | The above will generate the functions 'sexp_of_t' and 't_of_sexp'. The |
| 87 | preprocessor also supports automatic addition of conversion functions to |
| 88 | signatures. Just add 'with sexp' to the type in a signature, and the |
| 89 | appropriate function signatures will be generated. |
| 90 | |
| 91 | Converters for standard types (int, list, Hashtbl.t, etc.) become visible to |
| 92 | the macro-generated code by opening the standard module before their first use |
| 93 | in a type definition. Users will therefore usually want to place the following |
| 94 | at the top of their files: |
| 95 | << open Sexplib.Std |
| 96 | >> |
| 97 | |
| 98 | See the file 'lib_test/conv_test.ml' for example usage. It also demonstrates |
| 99 | how to extract and substitute sub-expressions. |
| 100 | |
| 101 | |
| 102 | 3.1 Compiling and linking |
| 103 | ========================== |
| 104 | |
| 105 | To compile a file you will have to add a preprocessing flag to the compiler |
| 106 | invocation, e.g. for foo.ml: |
| 107 | << ocamlc -pp "camlp4o -I {path to type_conv} \ |
| 108 | -I {path to sexplib} pa_type_conv.cmo pa_sexp_conv.cmo" \ |
| 109 | -I {path to sexplib} foo.ml |
| 110 | >> |
| 111 | |
| 112 | If you are using OCamlMakefile, just put the following line at the top of |
| 113 | the file, assuming you have installed both 'type_conv' and 'sexplib' with |
| 114 | ocamlfind. The comment must start at the beginning of the line, and you must |
| 115 | not break lines (here broken for readability only): |
| 116 | << (*pp camlp4o -I `ocamlfind query type_conv` \ |
| 117 | -I `ocamlfind query sexplib` \ |
| 118 | pa_type_conv.cmo pa_sexp_conv.cmo *) |
| 119 | >> |
| 120 | |
| 121 | Don't forget to place the macro 'TYPE_CONV_PATH', which takes a string |
| 122 | argument, at the top of the file to be preprocessed (see example |
| 123 | 'conv_test.ml'). It is supposed to set the module path to the given module |
| 124 | file. This is necessary, because modules may get packed at a later stage, and |
| 125 | error messages generated by Sexplib will refer to this location to help |
| 126 | pinpointing the associated type. |
| 127 | |
| 128 | In the linking stage you will only have to link with 'sexplib'. E.g. when |
| 129 | using OCamlMakefile just add it to the 'PACKS'-variable. |
| 130 | |
| 131 | |
| 132 | 4 Syntax Specification of S-expressions |
| 133 | *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*= |
| 134 | |
| 135 | |
| 136 | |
| 137 | |
| 138 | 4.1 Lexical conventions of S-expression |
| 139 | ======================================== |
| 140 | |
| 141 | Whitespace, which consists of space, newline, carriage return, horizontal |
| 142 | tab and form feed, is ignored unless within an OCaml-string, where it is |
| 143 | treated according to OCaml-conventions. The semicolon introduces comments. |
| 144 | Comments are ignored, and range up to the next newline character. The left |
| 145 | parenthesis opens a new list, the right parenthesis closes it again. Lists can |
| 146 | be empty. The double quote denotes the beginning and end of a string following |
| 147 | the lexical conventions of OCaml (see OCaml-manual for details). All |
| 148 | characters other than double quotes, left- and right parentheses, and |
| 149 | whitespace are considered part of a contiguous string. |
| 150 | |
| 151 | |
| 152 | 4.2 Grammar of S-expressions |
| 153 | ============================= |
| 154 | |
| 155 | S-expressions are either strings (= atoms) or lists. The lists can |
| 156 | recursively contain further S-expressions or be empty, and must be balanced, |
| 157 | i.e. parentheses must match. |
| 158 | |
| 159 | |
| 160 | 4.3 Examples |
| 161 | ============= |
| 162 | |
| 163 | << this_is_an_atom_123'&^%! ; this is a comment |
| 164 | "another atom in an OCaml-string \"string in a string\" \123" |
| 165 | |
| 166 | ; empty list follows below |
| 167 | () |
| 168 | |
| 169 | ; a more complex example |
| 170 | ( |
| 171 | ( |
| 172 | list in a list ; comment within a list |
| 173 | (list in a list in a list) |
| 174 | 42 is the answer to all questions |
| 175 | ) |
| 176 | ) |
| 177 | >> |
| 178 | |
| 179 | |
| 180 | |
| 181 | 4.4 Conversion of basic OCaml-values |
| 182 | ===================================== |
| 183 | |
| 184 | Basic OCaml-values like the unit-value, integers (in all representations), |
| 185 | floats, strings, and booleans are represented in S-exp syntax in the same way |
| 186 | as in OCaml. Strings may also appear without quotes if this does not clash |
| 187 | with the lexical conventions for S-expressions. |
| 188 | |
| 189 | |
| 190 | 4.5 Conversion of OCaml-tuples |
| 191 | =============================== |
| 192 | |
| 193 | OCaml-tuples are simple lists of values in the same order as in the tuple. |
| 194 | E.g.: |
| 195 | << (3.14, "foo", "bar bla", 27) <===> (3.14 foo "bar bla" 27) |
| 196 | >> |
| 197 | |
| 198 | |
| 199 | |
| 200 | 4.6 Conversion of OCaml-records |
| 201 | ================================ |
| 202 | |
| 203 | OCaml-records are represented as lists of pairs in S-expression syntax. Each |
| 204 | pair consists of the name of the record field (first element), and its value |
| 205 | (second element). E.g.: |
| 206 | << { |
| 207 | foo = 3; |
| 208 | bar = "some string"; |
| 209 | } |
| 210 | |
| 211 | <===> |
| 212 | |
| 213 | ( |
| 214 | (foo 3) |
| 215 | (bar "some string") |
| 216 | ) |
| 217 | >> |
| 218 | |
| 219 | Type specifications of records allow the use of a special type 'sexp_option' |
| 220 | which indicates that a record field should be optional. E.g.: |
| 221 | << type t = |
| 222 | { |
| 223 | x : int option; |
| 224 | y : int sexp_option; |
| 225 | } |
| 226 | >> |
| 227 | |
| 228 | The type 'sexp_option' is equivalent to ordinary options, but is treated |
| 229 | specially by the code generator. The above would lead to the following |
| 230 | equivalences of values and S-expressions: |
| 231 | << { |
| 232 | x = Some 1; |
| 233 | y = Some 2; |
| 234 | } |
| 235 | |
| 236 | <===> |
| 237 | |
| 238 | ( |
| 239 | (x (some 1)) |
| 240 | (y 2) |
| 241 | ) |
| 242 | >> |
| 243 | |
| 244 | And: |
| 245 | << { |
| 246 | x = None; |
| 247 | y = None; |
| 248 | } |
| 249 | |
| 250 | <===> |
| 251 | |
| 252 | ( |
| 253 | (x none) |
| 254 | ) |
| 255 | >> |
| 256 | |
| 257 | Note how 'sexp_option' allows you to leave away record fields that should |
| 258 | default to 'None'. It is also unnecessary (and actually wrong) now to write |
| 259 | down such a value as an option, i.e. the 'some'-tag must be dropped if the |
| 260 | field should be defined. |
| 261 | |
| 262 | The types 'sexp_list', 'sexp_array', and 'sexp_bool' can be used in ways |
| 263 | similar to the type 'sexp_option'. They assume the empty list, empty array, |
| 264 | and false value respectively as default values. |
| 265 | |
| 266 | |
| 267 | 4.7 Conversion of sum types |
| 268 | ============================ |
| 269 | |
| 270 | Constant constructors in sum types are represented as strings. Constructors |
| 271 | with arguments are represented as lists, the first element being the |
| 272 | constructor name, the rest being its arguments. Constructors may also be |
| 273 | started in lowercase in S-expressions, but will always be converted to |
| 274 | uppercase when converting from OCaml-values. |
| 275 | |
| 276 | For example: |
| 277 | << type t = A | B of int * float * t with sexp |
| 278 | |
| 279 | B (42, 3.14, B (-1, 2.72, A)) <===> (B 42 3.14 (B -1 2.72 A)) |
| 280 | >> |
| 281 | |
| 282 | The above example also demonstrates recursion in datastructures. |
| 283 | |
| 284 | |
| 285 | 4.8 Conversion of variant types |
| 286 | ================================ |
| 287 | |
| 288 | The conversion of polymorphic variants is almost the same as with sum types. |
| 289 | The notable difference is that variant constructors must always start with an |
| 290 | either lower- or uppercase character, matching the way it was specified in the |
| 291 | type definition. This is because OCaml also distinguishes between upper- and |
| 292 | lowercase variant constructors. Note that type specifications containing |
| 293 | unions of variant types are also supported by the S-expression converter. |
| 294 | |
| 295 | |
| 296 | 4.9 Conversion of OCaml-lists and arrays |
| 297 | ========================================= |
| 298 | |
| 299 | OCaml-lists and arrays are straightforwardly represented as S-expression |
| 300 | lists. |
| 301 | |
| 302 | |
| 303 | 4.10 Conversion of option types |
| 304 | ================================ |
| 305 | |
| 306 | The option type is converted like ordinary polymorphic sum types, i.e.: |
| 307 | << None <===> none |
| 308 | Some value <===> (some value) |
| 309 | >> |
| 310 | |
| 311 | There is a deprecated version of the syntax in which values of option type |
| 312 | are represented as lists in S-expressions: |
| 313 | << None <===> () |
| 314 | Some value <===> (value) |
| 315 | >> |
| 316 | |
| 317 | Reading of the old-style S-expression syntax for option values is only |
| 318 | supported if the reference 'Conv.read_old_option_format' is set to 'true' |
| 319 | (currently the default, which may change soon). A conversion exception is |
| 320 | raised otherwise. The old format will be written only if |
| 321 | 'Conv.write_old_option_format' is true (also currently the default). Reading |
| 322 | of the new format is always supported. |
| 323 | |
| 324 | |
| 325 | 4.11 Conversion of polymorphic values |
| 326 | ====================================== |
| 327 | |
| 328 | There is nothing special about polymorphic values as long as there are |
| 329 | conversion functions for the type parameters. E.g.: |
| 330 | << type 'a t = A | B of 'a with sexp |
| 331 | type foo = int t with sexp |
| 332 | >> |
| 333 | |
| 334 | In the above case the conversion functions will behave as if 'foo' had been |
| 335 | defined as a monomorphic version of 't' with ''a' replaced by 'int' on the |
| 336 | right hand side. |
| 337 | |
| 338 | If a datastructure is indeed polymorphic, and you want to convert it, you |
| 339 | will have to supply the conversion functions for the type parameters at |
| 340 | runtime. E.g. in the above example, if you wanted to convert a value of type |
| 341 | ''a t', you would have to write something like this: |
| 342 | << sexp_of_t sexp_of_a v |
| 343 | >> |
| 344 | |
| 345 | where 'sexp_of_a', which may also be named differently in this particular |
| 346 | case, is a function that converts values of type ''a' to an S-expression. |
| 347 | Types with more than one parameter require passing conversion functions for |
| 348 | those parameters in the order of their appearance on the left hand side of the |
| 349 | type definition. |
| 350 | |
| 351 | |
| 352 | 4.12 Conversion of abstract datatypes |
| 353 | ====================================== |
| 354 | |
| 355 | Of course, if you want to convert an abstract datatype to an S-expression, |
| 356 | you will have to roll your own conversion function, which should produce |
| 357 | values of type 'Sexp.t' directly. If, however, you want to make use of your |
| 358 | abstract type within definitions of other types, make sure that you call your |
| 359 | conversion function appropriately: it should be in the same scope as the |
| 360 | typename, and must be named 'sexp_of_{typename}'. |
| 361 | |
| 362 | |
| 363 | 4.13 Conversion of hashtables |
| 364 | ============================== |
| 365 | |
| 366 | Hashtables, which are abstract values in OCaml, are represented as |
| 367 | association lists, i.e. lists of key-value pairs, e.g.: |
| 368 | << ((foo 42) (bar 3)) |
| 369 | >> |
| 370 | |
| 371 | Reading in the above S-expression as hashtable mapping strings to integers |
| 372 | ('(string, int) Hashtbl.t') will map '"foo"' to 42 and '"bar"' to 3. |
| 373 | |
| 374 | Note that the order of elements in the list may matter, because duplicates |
| 375 | are kept: bindings will be inserted into the hashtable in order of appearence. |
| 376 | Therefore, the last binding of a key will be the "visible" one, the others are |
| 377 | "hidden". See the OCaml-documentation on hashtables for details. |
| 378 | |
| 379 | Note, too, that polymorphic equality may not hold between conversions. You |
| 380 | will have to use a function implementing logical equality for that purpose. |
| 381 | |
| 382 | |
| 383 | 4.14 Conversion of opaque values |
| 384 | ================================= |
| 385 | |
| 386 | Opaque values are ones for which we do not want to perform conversions. This |
| 387 | may be, because we do not have S-expression converters for them, or because we |
| 388 | do not want to apply them in a particular type context, e.g. if the resulting |
| 389 | S-expression should be printed out but without superfluous information. To |
| 390 | prevent the preprocessor from generating calls to converters, simply apply the |
| 391 | qualifier 'sexp_opaque' as if it were a type constructor, e.g.: |
| 392 | << type foo = int * stuff sexp_opaque with sexp |
| 393 | >> |
| 394 | |
| 395 | Thus, there is no need to specify converters for type 'stuff', and if there |
| 396 | are any, they will not be used in this particular context. Needless to say, it |
| 397 | is not possible to convert such an S-expression back to the original value. |
| 398 | Here is an example conversion: |
| 399 | << (42, some_stuff) ===> (42, <opaque>) |
| 400 | >> |
| 401 | |
| 402 | |
| 403 | |
| 404 | 4.15 Conversion of exceptions |
| 405 | ============================== |
| 406 | |
| 407 | S-expression converters for exceptions can be automatically registered using |
| 408 | the 'with sexp' macro, e.g.: |
| 409 | << module M = struct |
| 410 | exception Foo of int with sexp |
| 411 | end |
| 412 | >> |
| 413 | |
| 414 | Such exceptions will be translated in a similar way as sum types, but their |
| 415 | constructor will be prefixed with the fully qualified module path (here: |
| 416 | 'M.Foo') so as to be able to discriminate between them without problems. |
| 417 | The user can then easily convert an exception matching the above one to an |
| 418 | S-expression using 'Sexplib.Conv.sexp_of_exn'. User-defined conversion |
| 419 | functions can be registered, too, by calling 'Sexplib.Conv.add_exn_converter'. |
| 420 | This should make it very convenient for users to catch arbitrary exceptions |
| 421 | escaping their program and pretty-printing them, including all arguments, as |
| 422 | S-expressions. The library already contains mappings for all known exceptions |
| 423 | that can escape functions in the OCaml standard library. |
| 424 | |
| 425 | |
| 426 | 5 I/O and type conversions |
| 427 | *=*=*=*=*=*=*=*=*=*=*=*=*=* |
| 428 | |
| 429 | |
| 430 | There are multiple ways of performing I/O with S-expressions. If exact error |
| 431 | locations are required when type conversions fail, S-expressions need to be |
| 432 | parsed with location annotations. In most cases users may want to use |
| 433 | functions like e.g. 'load_sexp_conv' or 'load_sexp_conv_exn', which load |
| 434 | S-expressions from files and convert them. Only when conversions fail, the |
| 435 | file will be reparsed with annotations, which is slower, and type errors will |
| 436 | be reported accurately with file, line number, column, and file position. |
| 437 | |
| 438 | |
| 439 | 6 Contact information |
| 440 | *=*=*=*=*=*=*=*=*=*=*= |
| 441 | |
| 442 | |
| 443 | In the case of bugs, feature requests and similar, you can contact us here: |
| 444 | opensource@janestreet.com |
| 445 | Up-to-date information concerning this library should be available here: |
| 446 | http://www.janestreet.com/ocaml |
| 447 | Enjoy!! |
| 448 | |
| 449 | ----------------------------------------------------------------------------- |
| 450 | |
| 451 | This document was translated from LaTeX by HeVeA (2). |
| 452 | -------------------------------------- |
| 453 | |
| 454 | |
| 455 | (1) http://www.janestreet.com |
| 456 | |
| 457 | (2) http://hevea.inria.fr/index.html |