Modules
*******

[FIXME: somewhat babbling; should be reviewed by someone who understands modules, once the new module system is in place]

When programs become large, naming conflicts can occur when a function or global variable defined in one file has the same name as a function or global variable in another file.  Even just a similarity between function names can cause hard-to-find bugs, since a programmer might type the wrong function name.

The approach used to tackle this problem is called information encapsulation, which consists of packaging functional units into a given name space that is clearly separated from other name spaces.

The language features that allow this are usually called the module system because programs are broken up into modules that are compiled separately (or loaded separately in an interpreter).

Older languages, like C, have limited support for name space manipulation and protection.  In C a variable or function is public by default, and can be made local to a module with the static keyword.  But you cannot reference public variables and functions from another module with different names.

More advanced module systems have become a common feature in recently designed languages: ML, Python, Perl, and Modula 3 all allow the renaming of objects from a foreign module, so they will not clutter the global name space.

Scheme and modules
==================

Scheme, as defined in R4RS, does not have a module system at all.

Aubrey Jaffer, mostly to support his portable Scheme library SLIB, implemented a provide/require mechanism for many Scheme implementations. Library files in SLIB provide a feature, and when user programs require that feature, the library file is loaded in.

For example, the file random.scm in the SLIB package contains the line

     (provide 'random)

so to use its procedures, a user would type

     (require 'random)

and they would magically become available, but still have the same names!  So this method is nice, but not as good as a full-featured module system.

The Guile module system
=======================

In 1996 Tom Lord implemented a full-featured module system for Guile which allows loading Scheme source files into a private name space.

This module system is regarded as being rather idiosyncratic, and will probably change to something more like the ML module system, so for now I will simply describe how it works for a couple of simple cases.

First of all, the Guile module system sets up a hierarchical name space, and that name space can be represented like Unix pathnames preceded by a # character.  The root name space for all Guile-supplied modules is called ice-9.

So for example, the SLIB interface, contained in $srcdir/ice-9/slib.scm, starts out with

     (define-module (ice-9 slib))

and a user program can use

     (use-modules (ice-9 slib))

to have access to all procedures and variables defined within the slib module with (define-public ...).

So here are the functions involved:
 -- syntax: define-module module-specification
     MODULE-SPECIFICATION is of the form (hierarchy file).  One example of this is

          (use-modules (ice-9 slib))

     define-module makes this module available to Guile programs under the given MODULE-SPECIFICATION.

 -- syntax: define-public ...
     Makes a procedure or variable available to programs that use the current module.

 -- syntax: use-modules module-specification
     MODULE-SPECIFICATION is of the form (hierarchy file).  One example of this is

          (use-modules (ice-9 slib))

     use-modules allows the current Guile program to use all publicly defined procedures and variables in the module denoted by MODULE-SPECIFICATION.

[FIXME: must say more, and explain, and also demonstrate a private name space use, and demonstrate how one would do Python's "from Tkinter import *" versus "import Tkinter".  Must also add something about paths and standards for contributed modules.]

 -- primitive: standard-eval-closure module
     Return an eval closure for the module MODULE.

Some modules are included in the Guile distribution; here are references to the entries in this manual which describe them in more detail:

boot-9
     boot-9 is Guile's initialization module, and it is always loaded when Guile starts up.

(ice-9 debug)
     Mikael Djurfeldt's source-level debugging support for Guile (see Debugger User Interface).

(ice-9 threads)
     Guile's support for multi threaded execution (see Scheduling).

(ice-9 slib)
     This module contains hooks for using Aubrey Jaffer's portable Scheme library SLIB from Guile (see SLIB).

(ice-9 jacal)
     This module contains hooks for using Aubrey Jaffer's symbolic math packge Jacal from Guile (see JACAL).

Dynamic Libraries
=================

Often you will want to extend Guile by linking it with some existing system library.  For example, linking Guile with a curses or termcap library would be useful if you want to implement a full-screen user interface for a Guile application.  However, if you were to link Guile with these libraries at compile time, it would bloat the interpreter considerably, affecting everyone on the system even if the new libraries are useful only to you.  Also, every time a new library is installed, you would have to reconfigure, recompile and relink Guile merely in order to provide a new interface.

Many Unix systems permit you to get around this problem by using dynamic loading.  When a new library is linked, it can be made a dynamic library by passing certain switches to the linker.  A dynamic library does not need to be linked with an executable image at link time; instead, the executable may choose to load it dynamically at run time.  This is a powerful concept that permits an executable to link itself with almost any library without reconfiguration, if it has been written properly.

Guile's dynamic linking functions make it relatively easy to write a module that incorporates code from third-party object code libraries.

 -- primitive: dynamic-link fname
     Open the dynamic library LIBRARY-FILE.  A library handle representing the opened library is returned; this handle should be used as the LIB argument to the following functions.

 -- primitive: dynamic-object? obj
     Return #t if OBJ is a dynamic library handle, or #f otherwise.

 -- primitive: dynamic-unlink dobj
     Unlink the library represented by LIBRARY-HANDLE, and remove any imported symbols from the address space. GJB:FIXME:DOC: 2nd version below: Unlink the indicated object file from the application.  The argument DYNOBJ must have been obtained by a call to dynamic-link.  After dynamic-unlink has been called on DYNOBJ, its content is no longer accessible.

 -- primitive: dynamic-func name dobj
     Search the dynamic object DOBJ for the C function indicated by the string NAME and return some Scheme handle that can later be used with dynamic-call to actually call the function.

     Regardless whether your C compiler prepends an underscore _ to the global names in a program, you should not include this underscore in FUNCTION.  Guile knows whether the underscore is needed or not and will add it when necessary.

 -- primitive: dynamic-call func dobj
     Call LIB-THUNK, a procedure of no arguments.  If LIB-THUNK is a string, it is assumed to be a symbol found in the dynamic library LIB and is fetched with dynamic-func.  Otherwise, it should be a function handle returned by a previous call to dynamic-func. The return value is unspecified. GJB:FIXME:DOC 2nd version below Call the C function indicated by FUNCTION and DYNOBJ.  The function is passed no arguments and its return value is ignored.  When FUNCTION is something returned by dynamic-func, call that function and ignore DYNOBJ.  When FUNCTION is a string (or symbol, etc.), look it up in DYNOBJ; this is equivalent to

          (dynamic-call (dynamic-func FUNCTION DYNOBJ #f))

     Interrupts are deferred while the C function is executing (with SCM_DEFER_INTS/SCM_ALLOW_INTS).

 -- primitive: dynamic-args-call func dobj args
     Call PROC, a dynamically loaded function, passing it the argument list ARGS (a list of strings).  As with dynamic-call, PROC should be either a function handle or a string, in which case it is first fetched from LIB with dynamic-func.

     PROC is assumed to return an integer, which is used as the return value from dynamic-args-call.

     GJB:FIXME:DOC 2nd version below Call the C function indicated by FUNCTION and DYNOBJ, just like dynamic-call, but pass it some arguments and return its return value.  The C function is expected to take two arguments and return an int, just like main:

          int c_func (int argc, char **argv);

     The parameter ARGS must be a list of strings and is converted into an array of char *.  The array is passed in ARGV and its size in ARGC.  The return value is converted to a Scheme number and returned from the call to dynamic-args-call.

 -- primitive: c-registered-modules
     Return a list of the object code modules that have been imported into the current Guile process.  Each element of the list is a pair whose car is the name of the module, and whose cdr is the function handle for that module's initializer function.  The name is the string that has been passed to scm_register_module_xxx.

 -- primitive: c-clear-registered-modules
     Destroy the list of modules registered with the current Guile process. The return value is unspecified.  Warning: this function does not actually unlink or deallocate these modules, but only destroys the records of which modules have been loaded.  It should therefore be used only by module bookkeeping operations.

[FIXME: provide a brief example here of writing the C hooks for an object code module, and using dynamic-link and dynamic-call to load the module.]

Dynamic Linking from Marius
===========================

Most modern Unices have something called shared libraries.  This ordinarily means that they have the capability to share the executable image of a library between several running programs to save memory and disk space.  But generally, shared libraries give a lot of additional flexibility compared to the traditional static libraries.  In fact, calling them `dynamic' libraries is as correct as calling them `shared'.

Shared libraries really give you a lot of flexibility in addition to the memory and disk space savings.  When you link a program against a shared library, that library is not closely incorporated into the final executable.  Instead, the executable of your program only contains enough information to find the needed shared libraries when the program is actually run.  Only then, when the program is starting, is the final step of the linking process performed.  This means that you need not recompile all programs when you install a new, only slightly modified version of a shared library.  The programs will pick up the changes automatically the next time they are run.

Now, when all the necessary machinery is there to perform part of the linking at run-time, why not take the next step and allow the programmer to explicitly take advantage of it from within his program?  Of course, many operating systems that support shared libraries do just that, and chances are that Guile will allow you to access this feature from within your Scheme programs.  As you might have guessed already, this feature is called dynamic linking (Some people also refer to the final linking stage at program startup as `dynamic linking', so if you want to make yourself perfectly clear, it is probably best to use the more technical term dlopening, as suggested by Gordon Matzigkeit in his libtool documentation.)

As with many aspects of Guile, there is a low-level way to access the dynamic linking apparatus, and a more high-level interface that integrates dynamically linked libraries into the module system.

Low level dynamic linking
-------------------------

When using the low level procedures to do your dynamic linking, you have complete control over which library is loaded when and what get's done with it.

 -- primitive: dynamic-link library
     Find the shared library denoted by LIBRARY (a string) and link it into the running Guile application.  When everything works out, return a Scheme object suitable for representing the linked object file. Otherwise an error is thrown.  How object files are searched is system dependent.

     Normally, LIBRARY is just the name of some shared library file that will be searched for in the places where shared libraries usually reside, such as in /usr/lib and /usr/local/lib.

 -- primitive: dynamic-object? val
     Determine whether VAL represents a dynamically linked object file.

 -- primitive: dynamic-unlink dynobj
     Unlink the indicated object file from the application.  The argument DYNOBJ should be one of the values returned by dynamic-link.  When dynamic-unlink has been called on DYNOBJ, it is no longer usable as an argument to the functions below and you will get type mismatch errors when you try to.

 -- primitive: dynamic-func function dynobj
     Search the C function indicated by FUNCTION (a string or symbol) in DYNOBJ and return some Scheme object that can later be used with dynamic-call to actually call this function.  Right now, these Scheme objects are formed by casting the address of the function to long and converting this number to its Scheme representation.

     Regardless whether your C compiler prepends an underscore _ to the global names in a program, you should not include this underscore in FUNCTION.  Guile knows whether the underscore is needed or not and will add it when necessary.

 -- primitive: dynamic-call function dynobj
     Call the C function indicated by FUNCTION and DYNOBJ.  The function is passed no arguments and its return value is ignored.  When FUNCTION is something returned by dynamic-func, call that function and ignore DYNOBJ.  When FUNCTION is a string (or symbol, etc.), look it up in DYNOBJ; this is equivalent to

          (dynamic-call (dynamic-func FUNCTION DYNOBJ #f))

     Interrupts are deferred while the C function is executing (with SCM_DEFER_INTS/SCM_ALLOW_INTS).

 -- primitive: dynamic-args-call function dynobj args
     Call the C function indicated by FUNCTION and DYNOBJ, just like dynamic-call, but pass it some arguments and return its return value.  The C function is expected to take two arguments and return an int, just like main:

          int c_func (int argc, char **argv);

     The parameter ARGS must be a list of strings and is converted into an array of char *.  The array is passed in ARGV and its size in ARGC.  The return value is converted to a Scheme number and returned from the call to dynamic-args-call.

When dynamic linking is disabled or not supported on your system, the above functions throw errors, but they are still available.

Here is a small example that works on GNU/Linux:

     (define libc-obj (dynamic-link "libc.so"))
     libc-obj
     => #<dynamic-object "libc.so">
     (dynamic-args-call 'rand libc-obj '())
     => 269167349
     (dynamic-unlink libc-obj)
     libc-obj
     => #<dynamic-object "libc.so" (unlinked)>

As you can see, after calling dynamic-unlink on a dynamically linked library, it is marked as (unlinked) and you are no longer able to use it with dynamic-call, etc.  Whether the library is really removed from you program is system-dependent and will generally not happen when some other parts of your program still use it.  In the example above, libc is almost certainly not removed from your program because it is badly needed by almost everything.

The functions to call a function from a dynamically linked library, dynamic-call and dynamic-args-call, are not very powerful. They are mostly intended to be used for calling specially written initialization functions that will then add new primitives to Guile. For example, we do not expect that you will dynamically link libX11 with dynamic-link and then construct a beautiful graphical user interface just by using dynamic-call and dynamic-args-call.  Instead, the usual way would be to write a special Guile<->X11 glue library that has intimate knowledge about both Guile and X11 and does whatever is necessary to make them inter-operate smoothly.  This glue library could then be dynamically linked into a vanilla Guile interpreter and activated by calling its initialization function.  That function would add all the new types and primitives to the Guile interpreter that it has to offer.

From this setup the next logical step is to integrate these glue libraries into the module system of Guile so that you can load new primitives into a running system just as you can load new Scheme code.

There is, however, another possibility to get a more thorough access to the functions contained in a dynamically linked library.  Anthony Green has written libffi, a library that implements a foreign function interface for a number of different platforms.  With it, you can extend the Spartan functionality of dynamic-call and dynamic-args-call considerably.  There is glue code available in the Guile contrib archive to make libffi accessible from Guile.

Putting Compiled Code into Modules
----------------------------------

The new primitives that you add to Guile with gh_new_procedure or with any of the other mechanisms are normally placed into the same module as all the other builtin procedures (like display). However, it is also possible to put new primitives into their own module.

The mechanism for doing so is not very well thought out and is likely to change when the module system of Guile itself is revised, but it is simple and useful enough to document it as it stands.

What gh_new_procedure and the functions used by the snarfer really do is to add the new primitives to whatever module is the current module when they are called.  This is analogous to the way Scheme code is put into modules: the define-module expression at the top of a Scheme source file creates a new module and makes it the current module while the rest of the file is evaluated.  The define expressions in that file then add their new definitions to this current module.

Therefore, all we need to do is to make sure that the right module is current when calling gh_new_procedure for our new primitives. Unfortunately, there is not yet an easy way to access the module system from C, so we are better off with a more indirect approach.  Instead of adding our primitives at initialization time we merely register with Guile that we are ready to provide the contents of a certain module, should it ever be needed.

 -- Function: void scm_register_module_xxx (char *NAME, void (*INITFUNC)(void))
     Register with Guile that INITFUNC will provide the contents of the module NAME.

     The function INITFUNC should perform the usual initialization actions for your new primitives, like calling gh_new_procedure or including the file produced by the snarfer.  When INITFUNC is called, the current module is a newly created module with a name as indicated by NAME.  Each definition that is added to it will be automatically exported.

     The string NAME indicates the hierachical name of the new module. It should consist of the individual components of the module name separated by single spaces.  That is, the Scheme module name (foo bar), which is a list, should be written as "foo bar" for the NAME parameter.

     You can call scm_register_module_xxx at any time, even before Guile has been initialized.  This might be useful when you want to put the call to it in some initialization code that is magically called before main, like constructors for global C++ objects.

     An example for scm_register_module_xxx appears in the next section.

Now, instead of calling the initialization function at program startup, you should simply call scm_register_module_xxx and pass it the initialization function.  When the named module is later requested by Scheme code with use-modules for example, Guile will notice that it knows how to create this module and will call the initialization function at the right time in the right context.

Dynamic Linking and Compiled Code Modules
-----------------------------------------

The most interesting application of dynamically linked libraries is probably to use them for providing compiled code modules to Scheme programs.  As much fun as programming in Scheme is, every now and then comes the need to write some low-level C stuff to make Scheme even more fun.

Not only can you put these new primitives into their own module (see the previous section), you can even put them into a shared library that is only then linked to your running Guile image when it is actually needed.

An example will hopefully make everything clear.  Suppose we want to make the Bessel functions of the C library available to Scheme in the module (math bessel).  First we need to write the appropriate glue code to convert the arguments and return values of the functions from Scheme to C and back.  Additionally, we need a function that will add them to the set of Guile primitives.  Because this is just an example, we will only implement this for the j0 function, tho.

     #include <math.h>
     #include <guile/gh.h>

     SCM
     j0_wrapper (SCM x)
     {
       return gh_double2scm (j0 (gh_scm2double (x)));
     }

     void
     init_math_bessel ()
     {
       gh_new_procedure1_0 ("j0", j0_wrapper);
     }

We can already try to bring this into action by manually calling the low level functions for performing dynamic linking.  The C source file needs to be compiled into a shared library.  Here is how to do it on GNU/Linux, please refer to the libtool documentation for how to create dynamically linkable libraries portably.

     gcc -shared -o libbessel.so -fPIC bessel.c

Now fire up Guile:

     (define bessel-lib (dynamic-link "./libbessel.so"))
     (dynamic-call "init_math_bessel" bessel-lib)
     (j0 2)
     => 0.223890779141236

The filename ./libbessel.so should be pointing to the shared library produced with the gcc command above, of course.  The second line of the Guile interaction will call the init_math_bessel function which in turn will register the C function j0_wrapper with the Guile interpreter under the name j0.  This function becomes immediately available and we can call it from Scheme.

Fun, isn't it?  But we are only half way there.  This is what apropos has to say about j0:

     (apropos 'j0)
     -| the-root-module: j0     #<primitive-procedure j0>

As you can see, j0 is contained in the root module, where all the other Guile primitives like display, etc live.  In general, a primitive is put into whatever module is the current module at the time gh_new_procedure is called.  To put j0 into its own module named (math bessel), we need to make a call to scm_register_module_xxx.  Additionally, to have Guile perform the dynamic linking automatically, we need to put libbessel.so into a place where Guile can find it.  The call to scm_register_module_xxx should be contained in a specially named module init function.  Guile knows about this special name and will call that function automatically after having linked in the shared library.  For our example, we add the following code to bessel.c:

     void scm_init_math_bessel_module ()
     {
       scm_register_module_xxx ("math bessel", init_math_bessel);
     }

The general pattern for the name of a module init function is: scm_init_, followed by the name of the module where the individual hierarchical components are concatenated with underscores, followed by _module.  It should call scm_register_module_xxx with the correct module name and the appropriate initialization function.  When that initialization function will be called, a newly created module with the right name will be the current module so that all definitions that the initialization functions makes will end up in the correct module.

After libbessel.so has been rebuild, we need to place the shared library into the right place.  When Guile tries to autoload the (math bessel) module, it looks not only for a file called math/bessel.scm in its %load-path, but also for math/libbessel.so.  So all we need to do is to create a directory called math somewhere in Guile's %load-path and place libbessel.so there.  Normally, the current directory . is in the %load-path, so we just use that for this example.

     % mkdir maths
     % cd maths
     % ln -s ../libbessel.so .
     % cd ..
     % guile
     guile> (use-modules (math bessel))
     guile> (j0 2)
     0.223890779141236
     guile> (apropos 'j0)
     -| bessel: j0      #<primitive-procedure j0>

That's it!

Note that we used a symlink to make libbessel.so appear in the right spot.  This is probably not a bad idea in general.  The directories that the %load-path normally contains are supposed to contain only architecture independent files.  They are not really the right place for a shared library.  You might want to install the libraries somewhere below exec_prefix and then symlink to them from the architecture independent directory.  This will at least work on heterogenous systems where the architecture dependent stuff resides in the same place on all machines (which seems like a good idea to me anyway).