HTTP, the Web, and All That
===========================

When Guile started back in the mid-nineties, the GNU system was still focused on producing a good POSIX implementation.  This is why Guile's POSIX support is good, and has been so for a while.

But times change, and in a way these days the web is the new POSIX: a standard and a motley set of implementations on which much computing is done.  So today's Guile also supports the web at the programming language level, by defining common data types and operations for the technologies underpinning the web: URIs, HTTP, and XML.

It is particularly important to define native web data types.  Though the web is text in motion, programming the web in text is like programming with goto: muddy, and error-prone.  Most current security problems on the web are due to treating the web as text instead of as instances of the proper data types.

In addition, common web data types help programmers to share code.

Well.  That's all very nice and opinionated and such, but how do I use the thing?  Read on!

Universal Resource Identifiers
------------------------------

     (use-modules (web uri))

 A data type for Universal Resource Identifiers, as defined in RFC
 3986.

 -- Special Form: uri?

 -- Special Form: uri-scheme

 -- Special Form: uri-userinfo

 -- Special Form: uri-host

 -- Special Form: uri-port

 -- Special Form: uri-path

 -- Special Form: uri-query

 -- Special Form: uri-fragment

 -- Function: build-uri scheme [#:userinfo] [#:host] [#:port] [#:path] [#:query] [#:fragment] [#:validate?]
     Construct a URI object. If VALIDATE? is true, also run some consistency checks to make sure that the constructed URI is valid.

 -- Function: declare-default-port! scheme port
     Declare a default port for the given URI scheme.

     Default ports are for printing URI objects: a default port is not printed.

 -- Function: parse-uri string
     Parse STRING into a URI object. Returns #f if the string could not be parsed.

 -- Function: unparse-uri uri
     Serialize URI to a string.

 -- Function: uri-decode str [#:charset]
     Percent-decode the given STR, according to CHARSET.

     Note that this function should not generally be applied to a full URI string. For paths, use split-and-decode-uri-path instead. For query strings, split the query on & and = boundaries, and decode the components separately.

     Note that percent-encoded strings encode bytes, not characters. There is no guarantee that a given byte sequence is a valid string encoding. Therefore this routine may signal an error if the decoded bytes are not valid for the given encoding. Pass #f for CHARSET if you want decoded bytes as a bytevector directly.

 -- Function: uri-encode str [#:charset] [#:unescaped-chars]
     Percent-encode any character not in UNESCAPED-CHARS.

     Percent-encoding first writes out the given character to a bytevector within the given CHARSET, then encodes each byte as %HH, where HH is the hexadecimal representation of the byte.

 -- Function: split-and-decode-uri-path path
     Split PATH into its components, and decode each component, removing empty components.

     For example, "/foo/bar/" decodes to the two-element list, ("foo" "bar").

 -- Function: encode-and-join-uri-path parts
     URI-encode each element of PARTS, which should be a list of strings, and join the parts together with / as a delimiter.

The Hyper-Text Transfer Protocol
--------------------------------

     (use-modules (web http))

This module has a number of routines to parse textual representations of HTTP data into native Scheme data structures.

It tries to follow RFCs fairly strictly---the road to perdition being paved with compatibility hacks---though some allowances are made for not-too-divergent texts (like a quality of .2 which should be 0.2, etc).

 -- Special Form: header-decl?

 -- Special Form: make-header-decl

 -- Special Form: header-decl-sym

 -- Special Form: header-decl-name

 -- Special Form: header-decl-multiple?

 -- Special Form: header-decl-parser

 -- Special Form: header-decl-validator

 -- Special Form: header-decl-writer

 -- Function: lookup-header-decl name
     Return the HEADER-DECL object registered for the given NAME.

     NAME may be a symbol or a string. Strings are mapped to headers in a case-insensitive fashion.

 -- Function: declare-header! sym name [#:multiple?] [#:parser] [#:validator] [#:writer]
     Define a parser, validator, and writer for the HTTP header, NAME.

     PARSER should be a procedure that takes a string and returns a Scheme value. VALIDATOR is a predicate for whether the given Scheme value is valid for this header. WRITER takes a value and a port, and writes the value to the port.

 -- Function: read-header port
     Reads one HTTP header from PORT. Returns two values: the header name and the parsed Scheme value. May raise an exception if the header was known but the value was invalid.

     Returns #F for both values if the end of the message body was reached (i.e., a blank line).

 -- Function: parse-header name val
     Parse VAL, a string, with the parser for the header named NAME.

     Returns two values, the header name and parsed value. If a parser was found, the header name will be returned as a symbol. If a parser was not found, both the header name and the value are returned as strings.

 -- Function: valid-header? sym val
     Returns a true value iff VAL is a valid Scheme value for the header with name SYM.

 -- Function: write-header name val port
     Writes the given header name and value to PORT. If NAME is a symbol, looks up a declared header and uses that writer. Otherwise the value is written using DISPLAY.

 -- Function: read-headers port
     Read an HTTP message from PORT, returning the headers as an ordered alist.

 -- Function: write-headers headers port
     Write the given header alist to PORT. Doesn't write the final \r\n, as the user might want to add another header.

 -- Function: parse-http-method str [start] [end]
     Parse an HTTP method from STR. The result is an upper-case symbol, like GET.

 -- Function: parse-http-version str [start] [end]
     Parse an HTTP version from STR, returning it as a major-minor pair. For example, HTTP/1.1 parses as the pair of integers, (1 . 1).

 -- Function: parse-request-uri str [start] [end]
     Parse a URI from an HTTP request line. Note that URIs in requests do not have to have a scheme or host name. The result is a URI object.

 -- Function: read-request-line port
     Read the first line of an HTTP request from PORT, returning three values: the method, the URI, and the version.

 -- Function: write-request-line method uri version port
     Write the first line of an HTTP request to PORT.

 -- Function: read-response-line port
     Read the first line of an HTTP response from PORT, returning three values: the HTTP version, the response code, and the "reason phrase".

 -- Function: write-response-line version code reason-phrase port
     Write the first line of an HTTP response to PORT.

HTTP Requests
-------------

     (use-modules (web request))

 -- Special Form: request?

 -- Special Form: request-method

 -- Special Form: request-uri

 -- Special Form: request-version

 -- Special Form: request-headers

 -- Special Form: request-meta

 -- Special Form: request-port

 -- Function: read-request port [meta]
     Read an HTTP request from PORT, optionally attaching the given metadata, META.

     As a side effect, sets the encoding on PORT to ISO-8859-1 (latin-1), so that reading one character reads one byte. See the discussion of character sets in "HTTP Requests" in the manual, for more information.

 -- Function: build-request [#:method] [#:uri] [#:version] [#:headers] [#:port] [#:meta] [#:validate-headers?]
     Construct an HTTP request object. If VALIDATE-HEADERS? is true, the headers are each run through their respective validators.

 -- Function: write-request r port
     Write the given HTTP request to PORT.

     Returns a new request, whose request-port will continue writing on PORT, perhaps using some transfer encoding.

 -- Function: read-request-body/latin-1 r
     Reads the request body from R, as a string.

     Assumes that the request port has ISO-8859-1 encoding, so that the number of characters to read is the same as the request-content-length. Returns #f if there was no request body.

 -- Function: write-request-body/latin-1 r body
     Write BODY, a string encodable in ISO-8859-1, to the port corresponding to the HTTP request R.

 -- Function: read-request-body/bytevector r
     Reads the request body from R, as a bytevector. Returns #f if there was no request body.

 -- Function: write-request-body/bytevector r bv
     Write BODY, a bytevector, to the port corresponding to the HTTP request R.

 -- Function: request-accept request [default='()]
 -- Function: request-accept-charset request [default='()]
 -- Function: request-accept-encoding request [default='()]
 -- Function: request-accept-language request [default='()]
 -- Function: request-allow request [default='()]
 -- Function: request-authorization request [default=#f]
 -- Function: request-cache-control request [default='()]
 -- Function: request-connection request [default='()]
 -- Function: request-content-encoding request [default='()]
 -- Function: request-content-language request [default='()]
 -- Function: request-content-length request [default=#f]
 -- Function: request-content-location request [default=#f]
 -- Function: request-content-md5 request [default=#f]
 -- Function: request-content-range request [default=#f]
 -- Function: request-content-type request [default=#f]
 -- Function: request-date request [default=#f]
 -- Function: request-expect request [default='()]
 -- Function: request-expires request [default=#f]
 -- Function: request-from request [default=#f]
 -- Function: request-host request [default=#f]
 -- Function: request-if-match request [default=#f]
 -- Function: request-if-modified-since request [default=#f]
 -- Function: request-if-none-match request [default=#f]
 -- Function: request-if-range request [default=#f]
 -- Function: request-if-unmodified-since request [default=#f]
 -- Function: request-last-modified request [default=#f]
 -- Function: request-max-forwards request [default=#f]
 -- Function: request-pragma request [default='()]
 -- Function: request-proxy-authorization request [default=#f]
 -- Function: request-range request [default=#f]
 -- Function: request-referer request [default=#f]
 -- Function: request-te request [default=#f]
 -- Function: request-trailer request [default='()]
 -- Function: request-transfer-encoding request [default='()]
 -- Function: request-upgrade request [default='()]
 -- Function: request-user-agent request [default=#f]
 -- Function: request-via request [default='()]
 -- Function: request-warning request [default='()]

 -- Function: request-absolute-uri r [default-host] [default-port]

HTTP Responses
--------------

     (use-modules (web response))

 -- Special Form: response?

 -- Special Form: response-version

 -- Special Form: response-code

 -- Function: response-reason-phrase response
     Return the reason phrase given in RESPONSE, or the standard reason phrase for the response's code.

 -- Special Form: response-headers

 -- Special Form: response-port

 -- Function: read-response port
     Read an HTTP response from PORT, optionally attaching the given metadata, META.

     As a side effect, sets the encoding on PORT to ISO-8859-1 (latin-1), so that reading one character reads one byte. See the discussion of character sets in "HTTP Responses" in the manual, for more information.

 -- Function: build-response [#:version] [#:code] [#:reason-phrase] [#:headers] [#:port]
     Construct an HTTP response object. If VALIDATE-HEADERS? is true, the headers are each run through their respective validators.

 -- Function: extend-response r k v . additional
     Extend an HTTP response by setting additional HTTP headers K, V. Returns a new HTTP response.

 -- Function: adapt-response-version response version
     Adapt the given response to a different HTTP version. Returns a new HTTP response.

     The idea is that many applications might just build a response for the default HTTP version, and this method could handle a number of programmatic transformations to respond to older HTTP versions (0.9 and 1.0). But currently this function is a bit heavy-handed, just updating the version field.

 -- Function: write-response r port
     Write the given HTTP response to PORT.

     Returns a new response, whose response-port will continue writing on PORT, perhaps using some transfer encoding.

 -- Function: read-response-body/latin-1 r
     Reads the response body from R, as a string.

     Assumes that the response port has ISO-8859-1 encoding, so that the number of characters to read is the same as the response-content-length. Returns #f if there was no response body.

 -- Function: write-response-body/latin-1 r body
     Write BODY, a string encodable in ISO-8859-1, to the port corresponding to the HTTP response R.

 -- Function: read-response-body/bytevector r
     Reads the response body from R, as a bytevector. Returns #f if there was no response body.

 -- Function: write-response-body/bytevector r bv
     Write BODY, a bytevector, to the port corresponding to the HTTP response R.

 -- Function: response-accept-ranges response [default=#f]
 -- Function: response-age response [default='()]
 -- Function: response-allow response [default='()]
 -- Function: response-cache-control response [default='()]
 -- Function: response-connection response [default='()]
 -- Function: response-content-encoding response [default='()]
 -- Function: response-content-language response [default='()]
 -- Function: response-content-length response [default=#f]
 -- Function: response-content-location response [default=#f]
 -- Function: response-content-md5 response [default=#f]
 -- Function: response-content-range response [default=#f]
 -- Function: response-content-type response [default=#f]
 -- Function: response-date response [default=#f]
 -- Function: response-etag response [default=#f]
 -- Function: response-expires response [default=#f]
 -- Function: response-last-modified response [default=#f]
 -- Function: response-location response [default=#f]
 -- Function: response-pragma response [default='()]
 -- Function: response-proxy-authenticate response [default=#f]
 -- Function: response-retry-after response [default=#f]
 -- Function: response-server response [default=#f]
 -- Function: response-trailer response [default='()]
 -- Function: response-transfer-encoding response [default='()]
 -- Function: response-upgrade response [default='()]
 -- Function: response-vary response [default='()]
 -- Function: response-via response [default='()]
 -- Function: response-warning response [default='()]
 -- Function: response-www-authenticate response [default=#f]

Web Handlers
------------

from request to response

Web Server
----------

(web server) is a generic web server interface, along with a main loop implementation for web servers controlled by Guile.

The lowest layer is the <server-impl> object, which defines a set of hooks to open a server, read a request from a client, write a response to a client, and close a server.  These hooks -- open, read, write, and close, respectively -- are bound together in a <server-impl> object.  Procedures in this module take a <server-impl> object, if needed.

A <server-impl> may also be looked up by name.  If you pass the http symbol to run-server, Guile looks for a variable named http in the (web server http) module, which should be bound to a <server-impl> object.  Such a binding is made by instantiation of the define-server-impl syntax.  In this way the run-server loop can automatically load other backends if available.

The life cycle of a server goes as follows:

 1. The open hook is called, to open the server. open takes 0 or more arguments, depending on the backend, and returns an opaque server socket object, or signals an error.

 2. The read hook is called, to read a request from a new client. The read hook takes one arguments, the server socket.  It should return three values: an opaque client socket, the request, and the request body. The request should be a <request> object, from (web request).  The body should be a string or a bytevector, or #f if there is no body.

     If the read failed, the read hook may return #f for the client socket, request, and body.

 3. A user-provided handler procedure is called, with the request and body as its arguments.  The handler should return two values: the response, as a <response> record from (web response), and the response body as a string, bytevector, or #f if not present.  We also allow the reponse to be simply an alist of headers, in which case a default response object is constructed with those headers.

 4. The write hook is called with three arguments: the client socket, the response, and the body.  The write hook returns no values.

 5. At this point the request handling is complete. For a loop, we loop back and try to read a new request.

 6. If the user interrupts the loop, the close hook is called on the server socket.

 -- Special Form: define-server-impl name open read write close

 -- Function: lookup-server-impl impl
     Look up a server implementation. If IMPL is a server implementation already, it is returned directly. If it is a symbol, the binding named IMPL in the (web server IMPL) module is looked up. Otherwise an error is signaled.

     Currently a server implementation is a somewhat opaque type, useful only for passing to other procedures in this module, like read-client.

 -- Function: open-server impl open-params
     Open a server for the given implementation. Returns one value, the new server object. The implementation's open procedure is applied to OPEN-PARAMS, which should be a list.

 -- Function: read-client impl server
     Read a new client from SERVER, by applying the implementation's read procedure to the server. If successful, returns three values: an object corresponding to the client, a request object, and the request body. If any exception occurs, returns #f for all three values.

 -- Function: handle-request handler request body state
     Handle a given request, returning the response and body.

     The response and response body are produced by calling the given HANDLER with REQUEST and BODY as arguments.

     The elements of STATE are also passed to HANDLER as arguments, and may be returned as additional values. The new STATE, collected from the HANDLER's return values, is then returned as a list. The idea is that a server loop receives a handler from the user, along with whatever state values the user is interested in, allowing the user's handler to explicitly manage its state.

 -- Function: sanitize-response request response body
     "Sanitize" the given response and body, making them appropriate for the given request.

     As a convenience to web handler authors, RESPONSE may be given as an alist of headers, in which case it is used to construct a default response. Ensures that the response version corresponds to the request version. If BODY is a string, encodes the string to a bytevector, in an encoding appropriate for RESPONSE. Adds a content-length and content-type header, as necessary.

     If BODY is a procedure, it is called with a port as an argument, and the output collected as a bytevector. In the future we might try to instead use a compressing, chunk-encoded port, and call this procedure later, in the write-client procedure. Authors are advised not to rely on the procedure being called at any particular time.

 -- Function: write-client impl server client response body
     Write an HTTP response and body to CLIENT. If the server and client support persistent connections, it is the implementation's responsibility to keep track of the client thereafter, presumably by attaching it to the SERVER argument somehow.

 -- Function: close-server impl server
     Release resources allocated by a previous invocation of open-server.

 -- Function: serve-one-client handler impl server state
     Read one request from SERVER, call HANDLER on the request and body, and write the response to the client. Returns the new state produced by the handler procedure.

 -- Function: run-server handler [impl] [open-params] . state
     Run Guile's built-in web server.

     HANDLER should be a procedure that takes two or more arguments, the HTTP request and request body, and returns two or more values, the response and response body.

     For example, here is a simple "Hello, World!" server:

           (define (handler request body)
             (values '((content-type . ("text/plain")))
                     "Hello, World!"))
           (run-server handler)

     The response and body will be run through sanitize-response before sending back to the client.

     Additional arguments to HANDLER are taken from STATE. Additional return values are accumulated into a new STATE, which will be used for subsequent requests. In this way a handler can explicitly manage its state.

     The default server implementation is http, which accepts OPEN-PARAMS like (#:port 8081), among others. See "Web Server" in the manual, for more information.

     (use-modules (web server))